Send email at the end of a series of long-running tasks (Huey, Celery)

This may be a silly question, but async code hurts my brain, so I can’t quite figure this out.

I’ve got the following code (abbreviated here) working with Huey, but it sends the email immediately and always returns true whether or not it needs to create a new Book.

(If it makes any difference, I’m running this in the same Fly.io machine as the Django code, not using a “worker” process or anything. All this async stuff is very new to me. :sweat_smile:)

Is there any way to hook into a callback or something when all the tasks have completed?

@db_task()
def import_single_book(row, user_id):
    user = get_object_or_404(User, id=user_id)
    # …
    book, created_book = Book.objects.get_or_create(
        title=row["Title"],
        user=user,
    )
    # …
    if created_book:
        return True


@db_task()
def import_from_goodreads(data, user_id):
    user = get_object_or_404(User, id=user_id)
    count = 0

    for row in data:
        created = import_single_book(row, user_id)

        if created:
            count += 1

    # Send an email to the user when import is done
    user.email_user(
        subject="Book Stacks import finished!",
        message=f"Your import of {count} books from Goodreads is done",
    )

    return f"{count} books imported"

I figured out it (thanks to a couple of comments from the Huey author on GitHub: 1, 2)!

tasks.py:

@db_task()
def import_from_goodreads(data, user_id):
    user = get_object_or_404(User, id=user_id)

    tasks = [import_single_book(row, user_id) for row in data]
    results = [task_result.get(blocking=True) for task_result in tasks]

    user.email_user(
        subject="Book Stacks import finished!",
        message=f"Your import of {results.count(True)} books from Goodreads is done",
    )

Then I needed to increase the number of “workers” in the Huey settings to more than 1:

settings.py:

HUEY = {
    "huey_class": "huey.SqliteHuey",
    "immediate": False,
    "consumer": {
        "workers": 2,
    },
}