Tasks framework versatility & performance

Django Internals
4

Hi @theorangeone,

Thanks for taking your time for the detailed response and providing a little history deep dive.

I hope you can tell that I am a big fan of the work you did and tasks in Django. I think many would agree that Django hasn’t seen such a big new feature since ASGI. :1st_place_medal:

Community adoption

100%, it’s going to take a while, especially since there are very mature options like Celery.
That being said, I secretly dream of a day when we have a Redis queue in Django. :slight_smile:

Dataclasses

Again, I do like dataclasses, but since the tasks implementation is swappable, the inheritance quirks are real. My main concern is that you can’t override fields, e.g., with properties, but it’s a small concern.

This isn’t quite true. Internally it’s mutated, namely so refresh functions as expected, and because in some cases it’s cleaner than passing everything into the constructor (sure, these might be solved by dropping dataclass ). However externally, it’s intended to be considered read-only (subtly different to immutable). Pure immutability wasn’t really a design goal.

I see… hm… I don’t think “internally mutable” but “externally immutable” communicates clearly to users. Especially since refresh will update in place, which even for a user makes the object mutable and requires state management for users.

Maybe it’s helpful if we flip a coin and decide on either paradigm.

about the changes…

  1. Cool, I can open an issue there. I thought more about cleanup on Trac, but I am not in a rush.
  2. typing.NamedTuple: attrs distinction didn’t age well. They are typed now. I also disagree with other points, since they focus on usage, not Python’s internal implementation. A tuple is not an object, which makes a big difference for performance. And honestly, this is my only angel. I want the have a smaller memory footprint and quicker serialization. Both are strong suits of a tuple. Yes, they are actually immutable, but as you mentioned before, this might be useful.
  3. Same as 1., but no rush.
  4. This is mainly about providing a base implementation for the magic methods, since the base task does already provide a priority. 3rd party packages will of course need to expand on this.
  5. Oh, ignore my comment then. I thought they were public.

Performance receipts

I am working on a new queue-agnostic worker pool (*like billiard in Celery with a Gunicorn interface): GitHub - codingjoe/threadmill: A queue agnostic worker for Django's task framework. · GitHub
This involved numerous benchmarks. I will try to create some reproducible benchmarks that underline some of my performance concerns.

3 Likes