Feedback regarding index_together deprecation

Django Internals
1

Currently Django 3.2 and Django 4.2 are LTS releases. As a maintainer on django-taggit, I try to support all of this across the board unless I really have a reason not to.

I want to preface this a bit by saying I obviously could have brought this up way earlier, but I hadn’t fully put the pieces together until today.

Django 4.2 reports a deprecation on index_together, suggesting instead to use indexes. Great.

index_together names the index in a DB-dependent way. This means that I can’t “just” hardcode the old name into migrations (that run across multiple machines). New Index objects demands a name. So if I want to update my migration file to use AddIndex for new deployments, I’m now hardcoding a “fake” name.

And of course we have migrations.RenameIndex as a nice helper. It can query the DB based on the field tuple to find the index (this doesn’t work if you’re using an in-memory DB, but that’s magic for you). RenameIndex does all this querying to figure out what the old index name is for a rename.

But RenameIndex is not available in Django 3.2. And the logic is subtle enough that I do not believe I can reproduce it well.

I would like to stop using deprecated stuff in 4.2 (and users have been complaining about this deprecation issue), but I have three options:

  • Tell Django 3.2 users “do the operation manually” while I ship a RenameIndex migration that only works in 4.x
  • Hold off on clearing these deprecation warnings until early 2024, Meanwhile this makes it harder for me to look at Django main
  • Remove support for 3.2 entirely (what I think I’m going to do here so I can move forward)

This is the first time in a really long time I felt like I couldn’t have my cake and eat it too regarding Django upgrades. I’ve usually felt like that (with some django version checks) I’ve always had a clean upgrade path that “just works”.

Here there’s the “original sin” of index_together not holding onto the index name in the migrations, but part of the cure for this (RenameIndex) ending up not being available in 3.2 was annoying.

I totally get wanting to get things onto deprecation paths, and I understand this case is super specific. But I wanted to share this as I spent a couple hours thinking about what is best here. In my magical ideal world, when a deprecation warning shows up, I should be able to update a library to no longer have that deprecation warning, ship across all supported Django releases at the time, and have as few django.VERSION checks as possible.

TL;DR: “remove index_together” implies "use indexes (and AddIndex(..., name=thing-that-was-db-dependent-that-no-longer-is).

This also implies using RenameIndex to actually get the index to be named the same across all environments in a new migration

When 4.2 deprecated index_together, 3.2 was still in the support window, but missed RenameIndex as a tool. This made it hard to go to 4.2 without simultaenously dropping support for 3.2 for reasons that felt incidental.

Thanks to everyone who works on this stuff, migrations are a nightmare to work on just at a conceptual level. And I know things are so much better than they used to be!

2

Hey @rtpg,

So, if it helps, I’m planning on dropping support for Django<4.2 over the autumn for all the packages I support (except perhaps django-appconf which is much used, and unchanged ≈forever).

4.1 is EOL on 5.0 final in December. 3.2 has another six months or so, but nobody still on it needs the latest version of anything. (The version they’re on doesn’t stop working because the new version updated.)

I think it’s reasonable to support (something like) The current major version of Django, and the latest LTS. WDYT?

3

If I am a user of Django 3.2 LTS today, what do I want to do for upgrading?

  • Upgrade all my dependencies to versions that support both 3.2 and 4.1 or 4.2 (I’d probably first target 4.1)
  • Deploy a release still on 3.2 but with these upgraded dependencies (perhaps piecemeal for trickier dependencies)
  • Then do the Django upgrade

On top of that, in that scenario, I often like to at least check out deprecation warnings. Cleaning them up is nice. Being able to clean them up without version gating is wonderful (not always doable and mostly an aesthetics thing).

Some people (not me!) Outright consider deprecation warnings as errors in their CI pipelines. They might be (as the kids say) mad lads/gals. But I get the feeling.

So as a person shipping a library that supports 3.2, I really wanted to make a release that would support 3.2 and 4.2 (while considering deprecation warnings as “error” for this meaning of support). I’ve already pushed a release that supports 4.2 while still using some deprecated stuff several months ago. But if you don’t want deprecation warnings then that release doesn’t “support” 4.2 (according to a bit of a silly definition, granted).

And even ignoring errors, I’m sure we’ve all worked on projects where we have perpetual logged warnings on django boot from some random lib (or Django itself!) using some deprecated method. Things boot fine but the default logging config is spitting stuff out. It’s aesthetics, but annoying.

As a library maintainer, my philosophy is that I would like to support the workflows of users who care about deprecation warnings. There might be situations where it’s just too much work(I think this index stuff is kind of that). But I try not to be “the library holding X back”, especially as changes are relatively straightforward.

Django itself is very good about these kinds of offramps for smooth deprecation. I am very glad for these when I use Django, because it means I don’t have the scary “unrollbackable upgrade deploy”. But in order for it to work at the Django level, Django libraries sometimes need to be a bit wider in support. Fortunately this is often very straightforward.

EDIT: this stuff is doubly tricky for migrations, because people don’t read release notes and then end up with downtime. Several Django libraries ship un-rollbackable data migrations at the same time as the code changes, and suddenly a user can’t rollback. Bit of a tarpit that’s hard to do much about, because it’s dependent on deployment state and we don’t really have that “in the code”, so to speak.

4

I think the promise here is that if you’re on an LTS and you run without warnings, you should be able to upgrade to the next LTS.

That doesn’t mean that you won’t have warnings once upgraded. Your job is to fix those in order to be able to upgrade to the next LTS (5.2 as will be).

I think as a library author that’s enough (for the first pass). Folks can silence the warnings to keep their CI happy, and open a tracking issue in their tracker.

You then update your package again, dropping Django 3.2, and fixing the warnings. They update, and close their tracking ticket.

:100: I think that’s admirable, and correct.

Running with -Werror in CI is great, but it has to be used with filterwarnings exactly for this kind of case.

(IMO :slight_smile: )

1 Like