Adding template fragments or partials for the DTL

Django Internals
1

I’d like to begin a discussion about adding some kind of inline partials or template fragments feature to the DTL.

For background, there’s an HTMX essay on this topic here: </> Htmx ~ Template Fragments

The basic idea is that you can re-use parts of a template, when just updating part of a page say.

Currently the DTL requires moving the reusable fragment into an include or a template tag, which is great in many cases, but can be a little heavyweight (certainly to begin). Being able to define these partials inline gives better Locality of Behaviour (a concept for which there’s another essay on the HTMX site).

There’s lots of history in the area — here’s one thread, that links to another, but there are many more similar ideas floated around — so there’s clearly some demand.

I gave a talk on this topic at DjangoCon Europe this year, and have put up a package called django-template-partials to help demonstrate the concept and drive discussion:

I’ll write up further thoughts, but will just note two points for now:

  1. There’s a similar package called django-render-block that’s in play, but:
    • I wanted partials to be reusable within a single template, which blocks are not.
    • I integrated with the template loader, so bar changing a template name, it’s transparent to the view layer, which means not rewriting view logic.
  2. The first-pass implementation required attaching a mapping to the template origin — in order to get info out of the parser. A good first step would be pondering if we can adjust the API there allow this in a cleaner way.

I think we should play with a few ideas before rushing into an implementation here for Django — no doubt there’ll be both good ideas and unforeseen problems. I think stating now that Django 5.1 at the earliest would allow the whole summer and more to experiment.

But I think this is something we should add now. So putting it out there to you :blush:

Thanks.

12 Likes
2

This will be very helpful.
To avoid browser caching problems, would it be possible, or wise, to automatically set the Vary header if a template has partials?

3

I don’t think that’s directly related @codetalcott. Caching considerations are orthogonal to the mechanism for generating the response body I think.

4

Hi Carlton,

I maintain the django-render-block package, I’m not sure if any of this could be done as modifications to that instead of spawning a separate package, but I’d be up for collaborating if you’re interested. Maybe both packages have their place, but wanted to reach out.

Integrating into the template loader sounds like a really interesting improvement! Overwriting views to call the separate render_block_to_string is quite annoying.

I haven’t yet used HTMX (although I find the idea very interesting) and have predominantly used django-render-block when I want to group together different renderings of related content (e.g. email subject, HTML content, and text content).

See also an issue filed about using HTMX worth django-render-block.

—Patrick

2 Likes
5

Hi @carltongibson, as said on DjangoCon this looks nice :slight_smile: What is not clear to me yet is how context passing to the partial works (I didn’t find an example in the tests, but I might just have missed it) – does it work in a for loop to render rows of a table etc?

Thinking more about tools like HTMX and Unpoly (https://demo.unpoly.com) they both need a way to render only part of the page. You have implemented that with your project and django-render-block seems similar enough. Personally I feel that django-render-block feels maybe a bit more natural. As an example let’s think about a template along the lines of:

<html>
  <head />
  <body>
    <div>
       <navigation>{% block navigation %} item 1, item 2, item 3{% endblock navigation %}</navigation>
       <main>
         {% block content %}This is my super duper content{% content %}
       </main>
    </div>
  </body>
</html>

With django-template-partials this would probably look like this:

{% startpartial "navigation" %}item 1, item 2, item 3{% endpartial %}
{% startpartial "content" %}This is my super duper content{% endpartial %}
<html>
  <head />
  <body>
    <div>
       <navigation>{% partial "navigation" %}</navigation>
       <main>
         {% partial "content" %}
       </main>
    </div>
  </body>
</html>

Both approaches get you to the same end-result and I guess it is a matter of taste what you like more (although as you said django-render-block doesn’t support reuse currently). So I guess there is just one thing missing: Integration into the template loader. Your approach of using #anchors is a good idea, people are used to anchors from HTML anyways, so that just feels natural. This leaves us with the question of “what” should be “registered” as targetable by anchors. For Django itself blocks certainly would make sense and if the template API is accommodating enough in that area I do not see a reason why 3rd party template tags shouldn’t be able to hook into that mechanism (so you could register your partials).

As for the partials itself, before we add something like that to Django we should look into how Jinja handles this with it’s macro tags. FWIW I could even imagine that we simply extend the {% block %}-tags to support reusing? Maybe via a {% block xyz hidden %} where the hidden attribute allows it to not render by default and then a {% call %} (naming is hard) to “call” a block.

Can’t wait to see a PR :wink:

Cheers,
Florian

2 Likes
6

Hi @clokep — thanks for posting.

I very much wanted to be able to leverage render-block, but couldn’t see a clear way to getting the partial (or block?) reuse that I needed. (The block lookup is also at-best linear complexity, whereas I wanted something constant time.)

I’d be very grateful for your input here: you’ll have learnt lessons that I won’t have spotted yet! The goal isn’t really a new library, but rather a change to the DTL itself so that neither are needed.

@apollo13 Thanks for your points…

I actually prefer the partials approach to blocks. I don’t know about your templates but, more times than not blocks in my templates are empty. Sometimes I put default content, and use block super, but more likely not. Adding a block around a bit to reuse with render-block works, but it’s a little artificial. For me, having the partials defined at the top makes the template structure easier to read, navigate, and edit, when I’m adjusting (say) just the partial. — but this whole paragraph is one we could back and forth on without resolve.

The partials approach though gives a very clean to-and-from using includes. Say I’m using includes now, I can merely move them to the top of my main template, and then adjust the {% include ... %} to {% partial ... %} and I’m done. The rest of my template is unaffected. Later, when Locality of Behaviour is no longer dominant, I can just do the reverse. Again, very little disruption to the rest of what I’m doing. It’s this clean growth path that I’d highlight when thinking about fit with the DTL.

I’m using partials already in complex templates, in loops etc. I’ve not hit any issues yet. A partial is ≈ a NodeList, which gets rendered with the context in play. There are some wiggles for when rendered separately, to account for binding to the request context, but that’s handled. We can add more tests happily.

Ref Jinja, I see this as entirely internal to the DTL… I’m not sure what’s relevant there: Django just provides the backend interface no? :thinking: (If adding features to Jinja is in scope for Django, that’s something that I’d prefer addressed separately, not least by someone who actually knows Jinja :sweat_smile:)

This ties into my point about needing to adjust the parsing API… — I wanted to use the ability to just set values in the context originally, but it turns out that doesn’t work, due to A) non-rendered nodes, declared outside a block, and B) ordering of rendering depending on template extensions — it’s just too fragile — so there needs to be someway (an official™ map to stash values in would be enough) to attach additional data to the template object from the parser. If we had that, I’d be able to clean up the initial implementation (which is pretty lightweight as it is) and we’d open up a pathway for more flexibility here. (My first pass would be to look at making this small change, before we settle the main topic here.)

Hope all that makes sense. Thanks both!
C.

7

Interesting, I guess I’ll have to play and try. In my naive thinking my blocks are always filled (granted not in the base template but let’s say in detail.html that extends base.html and then I would render detail.html#theblock)

Yes, as I said in the end (assuming all other things equal) it would be a matter of taste.

Good to know, I probably should have looked at the source.

I should have been more clear. I am all in for adding customizable partial/block targeting to the loader. But if we were to add {% partial %} itself to core (which I know you aren’t suggesting [yet]) it might make sense to see what features {% macro %} in jinja provides and if it were worth to add one or two of those to {% partial %} as well.

Do you think we can a) provide the API you need and b) also use this API to register {% block %}-tags by name in there so django-render-block could utilize it as well? Or are there to many differences between a custom template tag which ends up as a nodelist and a block (which iirc is somewhat special cased).

Just to be absolutely clear, if it wasn’t before: I’d love to have the needed parsing adjustments in core.

1 Like
8

Don’t know yet. Let me have a play :wink:

10

Hi @carltongibson. In the Usage section of your explanation of django-template-partials, it says:

define a re-usable partial at the top of your template

A question: Is it a requirement that the partial be defined at the top of the template? In keeping with the idea of locality of behavior, for me it would be more desirable to have the partial defined at the location in the template where it will be rendered (unless it is being used in more than one location in the same template). I have found that, more often than not, when I render a template fragment, it is only used one time in a single template. It is less often that I need to render the same fragment in more than one location in the template. And in the even more rare case that a fragment is rendered in more than one template, then I think it makes sense to pull that fragment out into another file and use an {% include %}.

I was not familiar with the htmx Template Fragments essay. The chill implementation described there seems to get at what I just described - keeping the fragment declaration where it occurs in the template. My feature request here had proposed the same thing except that I was thinking in terms of minimal additional code in the template file (I proposed using CSS selectors to find the fragment to be rendered and just including that as a parameter on the render_to_string method).

I appreciate this discussion and hope to see this feature in an upcoming Django release.

11

Thanks for starting this thread, I’d also be very pleased to have a good, htmx-friendly solution built in to the template language (without it being htmx-specific).

I also usually prefer to do “inline blocks” rather than “partials moved somewhere else”, and I’ve been using django-render-block. When it comes to htmx, it improves Locality of Behaviour, and usually improves the refactoring process when you first add htmx: in that situation, you are very often saying “instead of rendering the whole page, just re-render this part”. It’s neat that you can just wrap the bit to re-render in some tags, and not have to move it somewhere else, and when reading I find it easier to see how different parts of the page are going to update when they are physically in the place where they will be rendered.

This is a pattern that has proved very popular in my django-htmx-patterns site. The process is often:

  1. wrap part of the template in a block,
  2. add some hx- attributes
  3. add a single decorator to the view function, like @for_htmx(use_block_from_params=True).

I think in contrast to the patterns I saw Carlton present, I usually manage to keep all logic relating to the selection of which partial to use in the template itself, rather than having any names of partials in the view code. I was able to do this without integrating with the template loader, by leveraging TemplateResponse and hx-vals.

That said, there are times when I have a chunk of HTML or template that I would like to use multiple times in different places, and without having to creating a separate file, and then Carlton’s “partials” syntax is more attractive. Also in other cases where the contents of the partial are really large.

There are some other things I’m also missing, that are hard to do without proper integration into the template loader. In particular, “lazy” rendering:

Returning a TemplateResponse from a view works really great with htmx using my decorator method. A downside is that the decorator converts the lazy TemplateResponse to a rendered HttpResponse, which might not play well with other decorators or middleware that were expecting a TemplateResponse. It would be great if there was a way of telling TemplateResponse to just change which bit it was going to finally render. There may be complications with the hooks though.

2 Likes
12

Just on the inline vs separate partial question @ghickman is working on a PR to django-template-partials to add an inline arg to startpartial that would have it output in place, so one could have it either way.

1 Like
13

Yes, that’s an available pattern. The view would pull the partial to render from the request parameters, rather than knowing it directly :+1:

14

I am also up for imitating {% macro %}, maybe even using the same name to increase compatibility. I think the explicit arguments to macros could make them easier to use for partially rendering.

15

Looking quickly — I’m no Jinja expert! — Jinja’s macros look a lot like Django’s custom template tags. It looks like a big scope jump from what I’m initially suggesting here. (I’m not proposing to eliminate the need for the current includes, or writing your own template tags, or blocks, which are for inheritance as I see it.) :thinking:

1 Like
16

Thanks for kicking off this discussion Carlton!

It feels like we’re talking about two different topics here:

  1. a way to make sections of a template file addressable within the same template
  2. making blocks/tags addressable from the template loader

I haven’t had a use for repeating blocks of content where the original defintion isn’t rendered (which might be obvious from my ticket on django-template-partials), but it seems like something that would certainly be a useful tool for the DTL to include.

However, the more interesting part of django-template-partials, to me, is the ability to address a partial via the template loader. To my eye this fits with the linked HTMX article on fragments perfectly, and between django-template-partials and django-render-block there’s a clear use for it to the community.

Bonus third topic: I’ve not used {% macro %} but from the docs it looks a lot like the slippers components library, which would be great to have in core, but that’s definitely a discussion for another thread.

I’ve been pondering on this too, but I wonder if a different tag would help usability here? Block comes with lots of other functionality which might muddy the waters when a user is looking for “how to do partials in Django”?

17

I don’t think they look like inclusion tags or custom tags, they feel more like your partials. Ie compare:

{% macro foo() %}CHILD{% endmacro %}
{% block body %}{{ foo() }}{% endblock %}

with

{% startpartial "foo" %}CHILD{% endpartial %}
{% block body %}{% partial "foo" %}}{% endblock %}

These two would to the best of my knowledge have the same outcome. That is what I meant when I suggested to look into macros to see if they provided some further functionality that might make sense for partials (ie would it make sense to support {% partial "foo" key=value %} to customize the context?).

18

@apollo13 OK, gotcha. Will ponder :thinking:

I’m using with to do this currently. (I note include supports a with argument, but have to admit to never having used that, rather than the with tag, which I use all the time.)

19

Interesting. I always seem to forget the with-tag :slight_smile: So I guess we already have multiple options for passing stuff around at least as far as the include-tag is concerned. But this shouldn’t sidetrack us from the actual goal: Namely adding support for addressing template node lists via the loader. How to pass context to the partials then is independent of that.

1 Like
21

I’d love to see this happen! The lack of locally-defined template fragments has been one of my biggest annoyances with DTL. For me the best outcome here is if this could work by adding the rendered fragment in the context, so they are reusable with more flexibility than the current proposed {% partial %} or {% call %} API suggests. In particular, I’d want to be able to reuse my fragment/partial within another fragmentwithout requiring them both to be in the same file.

I think I’d recommend distinguishing between the fundamental need to be able to create and reuse template partials inline; and then once those fragments are defined; the need to reuse them in isolation to benefit from the performance / DX improvements of frameworks that support this concept like HTMX. Solving that first need would improve the HTMX locality of behaviour and what UI developers would call component composition a great deal. Responding to the latter need makes Django more compatible with that paradigm shift.

For inline fragments, re Jinja, I would say the closest match is block assignments with set. They look like this:

{% set navigation %}
    <li><a href="/">Index</a>
    <li><a href="/downloads">Downloads</a>
{% endset %}

<ol>{{ navigation }}</ol>

In the DTL world there are a few implementations of this. The closest to the above is {% fragment as variable_name %}{% endfragment %} from Slippers. IIRC it renders a NodeList based on the parent context, then adds the output as the desired variable name in the parent context.

For the second need to reuse those partials beyond their definition point, the closest I can think of is Jinja’s macros as already mentioned. I’m not sure there’s a built-in API to call them from Python but they’re callable within other templates with the import tag. Here, with a input macro inside forms.html:

{% import 'forms.html' as forms %}

{{ forms.input('username') }}

This is essentially using templates as modules with explicit import and export statements just like in lots of programming languages. Here, macros are callable so their context resolution probably works quite differently to what’s been proposed here so far but I guess in terms of developer experience it’s close.

22

Hey @thibaudcolas — thanks for the comments!

I think we should try to keep changes minimal, so that we can actually land them for one thing — AND I don’t want to turn the DTL into Jinja — just use Jinja is a legitimate position — BUT I think we should be able to get most of what you want…

  1. For adjusting the context for the partial I’m already using the {% with ... %} tag with zero issues.
  2. For capturing the partial for the context the next version (next week or so) of django-template-partials will have an inline argument to output in place (which folks have expressed a desire for). That should allow exactly combining with Slipper’s fragment tag to capture the output to the context. (Again a separate tag: I’d much rather combine simple tools that do one thing well that have monster tags that try and do everything.
  3. For reuse of fragments from other files, the template loader integration should let you just use a named partial with the existing {% include ... %} tag.

As I say, I’m prepping an update for django-template-partials now, and I will make sure to add examples of each of these to the docs there. I imagine more experimenting to do at that point. (After the Django 5.0 feature freeze, I’ll turn to the preliminary PR adjusting the DTL parser so at least we’re building off actual API here though.)