Django hangs on async views with asycio.gather and an async ORM call

[ras]

Hi everyone!

I’m coming here after opening ticket #34747 (Django hangs on async views with asycio.gather and an async ORM call) – Django and having it closed as #worksforme.

I have this simple view:

{{{
import asyncio

from django.http import HttpResponse

from myapp.models import MyModel

async def test_hang(request):
await asyncio.gather(MyModel.objects.acreate())
return HttpResponse(‘OK’)
}}}

that hangs when called from daphne or uvicorn. When called from the django shell (with ipdb) like await test_hang(None) it works with no problem.

I created this minimal view to reproduce the issue, but in my project I’m creating a lot of tasks that have long external API requests saving the results to the db. Notice that with only one task it already hangs.

I haven’t found anything in the async documentation / channels / daphne stating that I cannot do something like this.

When setting a trace with ipdb I found that the line that gets stuck is:

{{{

/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py:561
kev_list = self._selector.control(None, max_ev, timeout)
}}}

And when running with python -m ipdb and then ctrl c it gets stuck in this line:

{{{
File “/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py”, line 1132, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
}}}

Tried with Python 3.11.4 in OS X and in docker (alpine).

Shouldn’t this simple view be working? If not, we should document it somehow. Any pointers much appreciated!

Can someone with OS X please try it? That might be the problem.

Thanks!

Some more info that I didn’t include in the ticket:

I’m using postgresql.

This is my pip freeze:

aiohttp==3.8.4
aiopg==1.3.5
aiosignal==1.3.1
anyio==3.7.1
appnope==0.1.3
asgiref==3.7.2
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==23.1.0
autobahn==23.6.2
Automat==22.10.0
backcall==0.2.0
boto3==1.28.3
botocore==1.31.3
certifi==2023.5.7
cffi==1.15.1
channels==4.0.0
channels-postgres @ git+https://github.com/khamaileon/channels_postgres.git@b9ad94e5f0925dc6f87e68834bbcd78ea6c9956c
channels-redis==4.1.0
charset-normalizer==3.1.0
click==8.1.3
constantly==15.1.0
cryptography==41.0.1
daphne==4.0.0
dataclasses-json==0.5.8
decorator==5.1.1
defusedxml==0.7.1
Django==4.2.2
django-allauth==0.54.0
django-environ==0.10.0
django-extensions==3.2.3
django-jsonform==2.17.2
django-s3-storage==0.14.0
djangorestframework==3.14.0
executing==1.2.0
frozenlist==1.3.3
greenlet==2.0.2
h11==0.14.0
httptools==0.5.0
hyperlink==21.0.0
idna==3.4
incremental==22.10.0
ipdb==0.13.13
ipython==8.14.0
jedi==0.18.2
jmespath==1.0.1
json-stream==2.3.2
json-stream-rs-tokenizer==0.4.21
langchain==0.0.227
langchainplus-sdk==0.0.20
lmql==0.0.6.5
MarkupSafe==2.1.3
marshmallow==3.19.0
marshmallow-enum==1.5.1
matplotlib-inline==0.1.6
msgpack==1.0.5
multidict==6.0.4
mypy-extensions==1.0.0
nodeenv==1.8.0
numexpr==2.8.4
numpy==1.25.0
oauthlib==3.2.2
openai==0.27.8
openapi-schema-pydantic==1.2.4
packaging==23.1
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.38
psutil==5.9.5
psycopg2-binary==2.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==1.10.9
Pygments==2.15.1
PyJWT==2.7.0
pyOpenSSL==23.2.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python3-openid==3.2.0
pytz==2023.3
PyYAML==6.0
redis==4.6.0
regex==2023.6.3
requests==2.31.0
requests-oauthlib==1.3.1
s3transfer==0.6.1
service-identity==23.1.0
six==1.16.0
sniffio==1.3.0
SQLAlchemy==2.0.17
sqlparse==0.4.4
stack-data==0.6.2
tenacity==8.2.2
termcolor==2.3.0
tiktoken==0.4.0
tqdm==4.65.0
traitlets==5.9.0
Twisted==22.10.0
txaio==23.1.1
typing-inspect==0.9.0
typing_extensions==4.6.3
urllib3==1.26.16
uvicorn==0.22.0
uvloop==0.17.0
watchfiles==0.19.0
wcwidth==0.2.6
websockets==11.0.3
Werkzeug==2.3.6
yarl==1.9.2
zope.interface==6.0

This are my settings:

import environ
import os
from pathlib import Path

env = environ.Env(
    DEBUG=(bool, False)
)
environ.Env.read_env()

BASE_DIR = Path(__file__).resolve().parent.parent

SECRET_KEY = env('SECRET_KEY')

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': env('DATABASE_NAME'),
        'USER': env('DATABASE_USER'),
        'PASSWORD': env('DATABASE_PASSWORD'),
        'HOST': env('DATABASE_HOST'),
        'PORT': env('DATABASE_PORT'),
    },
    'channels_postgres': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': env('DATABASE_NAME'),
        'USER': env('DATABASE_USER'),
        'PASSWORD': env('DATABASE_PASSWORD'),
        'HOST': env('DATABASE_HOST'),
        'PORT': env('DATABASE_PORT'),
    },
}

REDIS_LLM_MEMORY = f"{ env('REDIS_URL') }0"
CHANNEL_LAYERS = {
    "default": {
        'BACKEND': 'channels_postgres.core.PostgresChannelLayer',
        'CONFIG': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': env('DATABASE_NAME'),
            'USER': env('DATABASE_USER'),
            'PASSWORD': env('DATABASE_PASSWORD'),
            'HOST': env('DATABASE_HOST'),
            'PORT': env('DATABASE_PORT'),
        },
    },
}

ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
CSRF_TRUSTED_ORIGINS = env.list('CSRF_TRUSTED_ORIGINS')

# Application definition

INSTALLED_APPS = [
    'daphne',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.sites',

    'rest_framework',
    'django_jsonform',
    'channels_postgres',

    'accounts',
    'evaluation',
    'interview',
    'utils',
    'front',

    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    'allauth.socialaccount.providers.google',
]

MIDDLEWARE = [
    'utils.middleware.HealthCheckMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

SITE_ID = 1

ROOT_URLCONF = 'myproject.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

# custom form renderer
from django.forms.renderers import TemplatesSetting

class CustomFormRenderer(TemplatesSetting):
    form_template_name = "forms/form_snippet.html"

FORM_RENDERER = "angelai.settings.CustomFormRenderer"

WSGI_APPLICATION = 'myproject.wsgi.application'
ASGI_APPLICATION = 'myproject.asgi.application'

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

AUTH_USER_MODEL = "accounts.User"

AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
    'allauth.account.auth_backends.AuthenticationBackend',
]

ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_CONFIRM_EMAIL_ON_GET = True
ACCOUNT_EMAIL_VERIFICATION = 'none'
ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'http'
ACCOUNT_MAX_EMAIL_ADDRESSES = 1
ACCOUNT_LOGIN_ATTEMPTS_LIMIT = 15
ACCOUNT_EMAIL_CONFIRMATION_ANONYMOUS_REDIRECT_URL = True
ACCOUNT_LOGOUT_ON_GET = True
ACCOUNT_LOGIN_ON_PASSWORD_RESET = True
ACCOUNT_PRESERVE_USERNAME_CASING = False
ACCOUNT_SESSION_REMEMBER = True
ACCOUNT_USERNAME_REQUIRED = False
SOCIALACCOUNT_AUTO_SIGNUP = True
SOCIALACCOUNT_LOGIN_ON_GET = True

# Provider specific settings
SOCIALACCOUNT_PROVIDERS = {
    'google': {
        # For each OAuth based provider, either add a ``SocialApp``
        # (``socialaccount`` app) containing the required client
        # credentials, or list them here:
        'APP': {
            'client_id': env('GOOGLE_OAUTH_CLIENT_ID'),
            'secret': env('GOOGLE_OAUTH_CLIENT_SECRET'),
            'key': ''
        }
    }
}

from django.urls import reverse_lazy
LOGIN_REDIRECT_URL = reverse_lazy('start')

# Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True


STATIC_URL = 'static/'
STATIC_ROOT = BASE_DIR / '../collected_static'
STATICFILES_DIRS = [
    ("frontend", BASE_DIR / "../frontend/dist"),
]
ALLOW_FRONTEND_DEV = False

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

OPENAI_API_KEY = env('OPENAI_API_KEY')
GOOGLE_ANALYTICS_ID=env('GOOGLE_ANALYTICS_ID')
EVALUATION_PITCHES_DIR = "../another"
EVALUATION_SCORE_TOLERANCE = 2
OPENAI_OPTIONS = {
    'model': 'gpt-3.5-turbo-16k',
    'temperature': .8,
    'verbose': True,
}

USE_TLS = False
HTTP_PROTOCOL = 'http'
WEBSOCKET_PROTOCOL = 'ws'

try:
    from myproject.settings_local import *
    INSTALLED_APPS.extend(EXTRA_INSTALLED_APPS)
except ImportError:
    # using print and not log here as logging is yet not configured
    print('local settings not found')
    pass

And my settings_local.py

BASE_DIR = Path(__file__).resolve().parent.parent

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

SITE_ID = 1

EXTRA_INSTALLED_APPS = (
    'django_extensions',
)

ALLOWED_HOSTS = ['*']

[ras]

I’ve fixed it. I’m writing my results here for anyone having this problem in the future.

I first created a bare bones project to test that the simple view worked. After that I started removing everything from my project till it started working.

My problem was that I had a middleware not prepared for async and that somehow broke this usage:

My middleware was this one:

from django.http import HttpResponse


class HealthCheckMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path == '/health':
            return HttpResponse('ok')
        return self.get_response(request)

and I changed it to this one and now it works.

from asgiref.sync import iscoroutinefunction

from django.http import HttpResponse
from django.utils.decorators import sync_and_async_middleware


@sync_and_async_middleware
def health_check_middleware(get_response):
    if iscoroutinefunction(get_response):

        async def middleware(request):
            if request.path == '/health':
                return HttpResponse('ok')
            response = await get_response(request)
            return response

    else:

        def middleware(request):
            if request.path == '/health':
                return HttpResponse('ok')
            response = get_response(request)
            return response

    return middleware

[doom]

@ras

My problem was that I had a middleware not prepared for async and that somehow broke this usage:

For reference about middle wares:

Django

[salomvary]

I had a similar problem, for me the troublemaker was whitenoise.middleware.WhiteNoiseMiddleware. Removing WhiteNoiseMiddleware fixed the problem.