StreamingHttpResponse must consume synchronous iterators

HI all :slight_smile: any ideas why i’m getting this?:

/config/.local/lib/python3.10/site-packages/django/http/response.py:534: Warning: StreamingHttpResponse must consume synchronous iterators in order to serve them asynchronously. Use an asynchronous iterator instead.

I cant see where either the setting for this is or why its happening randomly…

What version of Django are you using?

How are you running Django?

Are you running your views as asynchronous views? What views do you have that return a StreamingHttpResponse?

Hello again Ken… im using 4.2 and it is all running smashingly well in asgi async… i have never used StreamingHttpResponse, so i cant for the life of me figure out where it is coming from… It doesnt seem to be affecting any of my app, but it is one of those warnings i want to trace…

I run some views async def with async for on query loops.
hosted on azure.

A FileResponse is a subclass of a StreamingHttpResponse. Are you returning any FileResponse in your views?
Side note: This was updated as a result of #33735 (Add asynchronous responses for use with an ASGI server) – Django. There are a fair number of notes within that ticket that may be worth reading.

Yep, see the 4.2 release notes and the docs on streaming responses.

Essentially you need to give the response an async iterator if you’re serving under ASGI, then it will stream.

If you give it a sync iterator, it will consume it fully in order to send the response. This will still work, and was the behaviour up until 4.1, but it’s likely not what you intended.

1 Like

FileResponse itself is not async streamable since file IO is not async.

Not that i remember or know of… i did a site wide reference to either terms and found nothing. I read the link you provided but it has gone a bit over my head :frowning:

As I said… no FileResponse being used currently . I predominantly use JsonResponse()

Do you have Django serving media files maybe? :thinking:

There’s a streaming response (or file response) in play somewhere.

(Django’s static files handler suppresses this warning, since it’s only for development usage, and it’s not something we can work around currently.)

Not saying i found the solution, however, my celery wasnt running ( as a seperate issue) because of missing shell information… when i fixed it, the warning has not shown since… maybe its coincidence… do you think it may have something to do with celery? django celery beat perhaps?

No way to say, not directly…

Someone somewhere is sending, likely, a FileResponse, and the warning is triggering.

The warning is triggered after the view function returns so I would set the -W error flag and run in the debugger to examine the response (and its content) to see where it’s coming from.

Unless you’re hitting resource limits you could just live with it. (It’s essentially the 4.1 behaviour, just with an added notice that it’s not streaming as intended). Ultimately whatever is raising the error will need to adjust to an async iterator (if possible) or be served via WSGI in order to stream.

like this ?

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
import warnings # remove warnings after development

def main():
    """Run administrative tasks."""
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'qengine.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    warnings.filterwarnings('error') # Show warnings as errors
    main()

Well, I meant with the -W flag to the python command…

python -W error manage.py runserver

i get this

abc@vscuser1:/workspace/q-engine-exp$ python3.10 -W error manage.py runserver
Traceback (most recent call last):
  File "/workspace/q-engine-exp/manage.py", line 23, in <module>
    main()
  File "/workspace/q-engine-exp/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/config/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/config/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/config/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/config/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/config/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/q-engine-exp/qengine/settings.py", line 512, in <module>
    sentry_sdk.init(
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/hub.py", line 120, in _init
    client = Client(*args, **kwargs)  # type: ignore
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/client.py", line 106, in __init__
    self._init_impl()
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/client.py", line 140, in _init_impl
    self.integrations = setup_integrations(
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/integrations/__init__.py", line 109, in setup_integrations
    for integration_cls in iter_default_integrations(
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/integrations/__init__.py", line 41, in iter_default_integrations
    yield getattr(import_module(module), cls)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/config/.local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 38, in <module>
    from celery.app.trace import task_has_custom
  File "/config/.local/lib/python3.10/site-packages/celery/app/__init__.py", line 2, in <module>
    from celery import _state
  File "/config/.local/lib/python3.10/site-packages/celery/_state.py", line 15, in <module>
    from celery.utils.threads import LocalStack
  File "/config/.local/lib/python3.10/site-packages/celery/utils/__init__.py", line 16, in <module>
    from .nodenames import nodename, nodesplit, worker_direct
  File "/config/.local/lib/python3.10/site-packages/celery/utils/nodenames.py", line 6, in <module>
    from kombu.entity import Exchange, Queue
  File "/config/.local/lib/python3.10/site-packages/kombu/entity.py", line 7, in <module>
    from .serialization import prepare_accept_content
  File "/config/.local/lib/python3.10/site-packages/kombu/serialization.py", line 440, in <module>
    for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
  File "/config/.local/lib/python3.10/site-packages/kombu/utils/compat.py", line 82, in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 430, in get
    self._warn()
DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.

i did as u suggested and this happened, its difficult for me to ascertain what is the cause as the trace has various libraries mentioned and i dont personally use SelectableGroups …

Ok, so you’re hitting a totally different warning there. You’ll need to set up your filters to just error on the warning from the streaming response.