\
**Category:** Async\
**Created:** [May 19, 2023, 3:22am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017 "2023-05-19T03:22:50Z")\
**Posts on this page:** 11\
**Page:** 1
**Author:**  [@Chris-May](https://forum.djangoproject.com/u/Chris-May)\
**Post date:** [May 19, 2023, 3:22am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/1 "2023-05-19T03:22:50Z")
Hello, Djangonauts!
I’ve been excited to leverage streaming HTML in Django since I read the [impressive series of blog posts from Taylor Hunt](https://dev.to/tigt/making-the-worlds-fastest-website-and-other-mistakes-56na), where he detailed how he vastly improved the experience of a Fortune 20 website, [even on slow Android phones](https://assets.codepen.io/183091/all-5-at-once.handbraked.mp4) by using as little JavaScript as he could get away with and leveraging the old technology of streaming HTML.
I created a [small proof of concept](https://github.com/Chris-May/django_streaming_html_poc) of streaming HTML in Django. It renders a home page of an e-commerce website that acts like it recommends specific items for the current viewer. I added an `asyncio.sleep` call to mimic a slow database query or API call. Thanks to the streaming HTML, all of the page before that section renders quickly, giving the impression of a performant website.
I’m excited that this PoC shows it’s possible, but I’m a little disappointed with how I got it to work. I had to:
- render the shell, or base, HTML file that holds all the header content
- split it apart at a place where I knew I could insert the content
- create two additional templates to handle the parts of the page before and after the slow content
- then render a fourth template for the slow data
An excerpt of the [`views.py`](https://github.com/Chris-May/django_streaming_html_poc/blob/main/home/views.py) file is below:
```python
customized_recommendations = [
# example data
dict(name='Comfy Chair', discount_price=745.00, normal_price=800.00, review_count=1550, img='product1.jpg'),
...
]
async def recommendations():
# split the shell template to sandwich the rest of the HTML
pre_shell, post_shell = render_to_string('shell.html').split('')
yield pre_shell
yield render_to_string('home/home_pre.html')
for item in customized_recommendations:
await asyncio.sleep(.7) # Faking an expensive database query or slow API
yield render_to_string('home/_item.html', dict(recommendation=item))
yield render_to_string('home/home_post.html')
yield post_shell
async def index(request):
return StreamingHttpResponse(recommendations())
```
Compare that to a [similar implementation in Starlette / Jinja](https://github.com/Chris-May/starlette_streaming_html_poc), where I could use one template:
```python
async def homepage(request):
recommendations = [
dict(name='Comfy Chair', discount_price=745.00, normal_price=800.00, review_count=1550, img='product1.jpg'),
...
]
async def slow_recommendations():
# fake a slow query or API request
for r in recommendations:
delay = random.randint(0, 500) / 100
await asyncio.sleep(delay)
yield r
template = templates.get_template('index.html')
return StreamingResponse(
content=template.generate_async(
request=request,
recommendations=slow_recommendations(),
),
media_type='text/html',
)
```
Calton asked me to look into what [Jinja is doing with that `generate_async` method](https://github.com/pallets/jinja/blob/main/src/jinja2/environment.py#L1358). I think it uses an async event loop to iterate through the templates, sending chunks down the wire when they’re ready and looping in a holding pattern while the async data call resolves.
I don’t know if improving the developer experience in this way is feasible, but I call it to your attention to see what you think.
Thank you so much for all your work at making such an incredible platform!
---
**Author:**  [@andrewgodwin](https://forum.djangoproject.com/u/andrewgodwin)\
**Post date:** [May 19, 2023, 3:58pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/2 "2023-05-19T15:58:24Z")
This has always been one of my “white whales” with async - where we can finally close the loop on rendering, as it were.
As it stands, I think we’d have to go in and overhaul the template engine to accomplish this; async functions, iterators and generators are different enough that they can’t really be slotted in, and the way templates render is especially difficult to work with in this context.
I’ve not been in that codebase in a while, but the first step would be having someone who knows templating somewhat well go and learn it and establish if there’s a core node render loop we can async-iterate over in a similar way to Jinja.
---
**Author:**  [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [May 19, 2023, 4:04pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/3 "2023-05-19T16:04:14Z")
> [@andrewgodwin](#):
>
> I’ve not been in that codebase in a while…
I wouldn’t panic, it’s hasn’t changed 😜
(Sorry for the noise, couldn’t resist. 🙂 )
---
**Author:**  [@adamchainz](https://forum.djangoproject.com/u/adamchainz)\
**Post date:** [July 24, 2023, 8:20am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/4 "2023-07-24T08:20:56Z")
Just found this in my inbox. I think a good first step would be using Jinja’s streaming rendering working, and add any necessary general template functionality to make that easier to access. Then we could consider the much larger task of adding streaming rendering to Django’s template engine.
---
**Author:**  [@Chris-May](https://forum.djangoproject.com/u/Chris-May)\
**Post date:** [August 3, 2023, 7:16pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/5 "2023-08-03T19:16:43Z")
Hey Adam! I want to understand your thoughts.
Are you saying I should change the above proof of concept with the Jinja template engine?
Or are you identifying a strategy for moving forward with incorporating this into Django?
Thanks for either suggestion! 😃
---
**Author:**  [@adamchainz](https://forum.djangoproject.com/u/adamchainz)\
**Post date:** [August 7, 2023, 5:30am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/6 "2023-08-07T05:30:38Z")
A strategy to incorporate this into Django. Since Django supports multiple template backends, we can divide the problem into general backend support and Django Template Language (DTL) engine support. First, let’s add whatever template backend support is necessary to get Jinja’s streaming rendering working. Then, let’s consider DTL support, which is going to be a much larger task.
---
**Author:**  [@HamaBarhamou](https://forum.djangoproject.com/u/HamaBarhamou)\
**Post date:** [December 5, 2023, 11:43am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/7 "2023-12-05T11:43:48Z")
hi, where are you with this new feature to add to django. is it available?
---
**Author:**  [@sarahboyce](https://forum.djangoproject.com/u/sarahboyce)\
**Post date:** [January 10, 2024, 8:22am UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/8 "2024-01-10T08:22:12Z")
This came to mind when I was reviewing the list of proposed GSOC ideas: [SummerOfCode2024 – Django](https://code.djangoproject.com/wiki/SummerOfCode2024#Ideas)
Adam do you think this is a good candidate to be added to the GSOC suggested ideas?
---
**Author:**  [@Chris-May](https://forum.djangoproject.com/u/Chris-May)\
**Post date:** [January 10, 2024, 3:56pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/9 "2024-01-10T15:56:10Z")
> [@adamchainz](#):
>
> Divide the problem into general backend support and Django Template Language (DTL) engine support. First, let’s add whatever template backend support is necessary to get Jinja’s streaming rendering working.
I’ve found it surprisingly easy to incorporate Jinja streaming rendering working.
I updated the above proof of concept to incorporate Jinja’s template engine as a second engine and [returned a streaming response](https://github.com/Chris-May/django_streaming_html_poc/blob/main/home/views/gin.py):
```python
from django.http import StreamingHttpResponse
from django.template.loader import get_template
async def index(request):
template = get_template('home.html')
return StreamingHttpResponse(template.template.generate_async(
context=dict(recommendations=get_recommendations())
))
```
---
**Author:**  [@adamchainz](https://forum.djangoproject.com/u/adamchainz)\
**Post date:** [January 20, 2024, 11:08pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/10 "2024-01-20T23:08:17Z")
That’s nice @Chris-May ! Glad you got it working.
That code uses the `template` attribute of Django’s generic [`Template` class](https://docs.djangoproject.com/en/stable/topics/templates/#module-django.template.loader) to access the underlying Jinja template instance. It then calls the Jinja `generate_async` API.
The “general backend support” that I suggested would mean a public API in Django, on the generic `Template` class, like the existing `Template.render`. Perhaps it could be called `Template.render_streaming()`. It can fail for DTL for the time being.
On top, there could be `StreamingTemplateResponse`, `django.shortcuts.render_streaming()`, or whatever else might be useful.
@sarahboyce this first step is way too small for GSoC alone. Adding streaming rendering to DTL would require a bunch of experimentation but could work as a project for a dedicated student!
---
**Author:**  [@plopidou](https://forum.djangoproject.com/u/plopidou)\
**Post date:** [March 21, 2024, 4:32pm UTC](https://forum.djangoproject.com/t/is-it-feasible-to-improve-streaming-html-in-django/21017/11 "2024-03-21T16:32:37Z")
God I love this image so much
