Unpredictably, the settings.FORCE_SCRIPT_NAME is stripped from the output generated by "static" template tag

Using Django Deployment
1

protracted whine

I deploy a django site to run in Docker swarm on a corporate LAN. It appears to me that
(sometimes), after django generates html pages, something is removing part of the URL django template
generates. I am not at all sure it is a django issue, it might be something about our reverse proxy (but that doesn’t quite make sense from symptoms either).

I don’t know what to ask/investigate next.

About 50% of the time when I deploy, the .css “works”. Other times, the path to css in
the rendered page in incorrect. I “fix” the problem by redeploying without code change, and after a few tries everything’s fine, and it stays fine until next server restart or deploy, after which it’s fine roughly 50% of the time. The difference does not correspond to which node in the swam it runs on.

Below I show

  1. the symptom, and
  2. that when I investigate with manage.py shell, it seems like the template static tag is working as expected. I don’t know how to verify that
    django.templatetags.static.static is actually getting called. If it’s not, my assumptions
    are wrong.

Details:

1) the symptom

In my setup, the reverse proxy (nginx I think) finds the container by having
‘/bi-web-utils/daily’ preprended to root, and I set that in
settings.FORCE_SCRIPT_NAME.

My sometimes-it-works html includes:

<link rel="stylesheet" href="{% static 'tableau_explorer/header.css' %}">

Right now, “it’s not working” and my rendered page shows

<link rel="stylesheet" href="/static/tableau_explorer/popup.css"> (missing the FORCE_SCRIPT_NAME.`
And, I have debugging html

	<P>The url from evaluating static is <pre>
	  /static/tableau_explorer/header.css.</pre></P>
	  <P>What was predicted from view was: <pre>/static/tableau_explorer/header.css</pre></P>

The value “/static/tableau_explorer/header.css” is calculated in view by static('tableau_explorer/header.css') as below:

from django.templatetags.static import static

def index(request):
    checkbox_defaults = {'use_acct': True,
                         'use_corp': True,
                         'use_dev': False,
                }
    return render(request,
                  template_name="tableau_explorer/index.html",
                  # {**d, **e} returns the merged dictionaries d & e
                  context={
                  **{'workbook_search_form': IndexPageSearchForm(prefix='WB'),
                  'everything_search_form': IndexPageSearchForm(prefix='EV')  },
                      **checkbox_defaults,
                      **dict(predicted_url=static('tableau_explorer/header.css'))
                   })

2) the static method is getting the value I would expect if I examine it in the shell:

Inside Django shell of the running container everything looks like I would expect it to look:

>>> from django.templatetags import static
>>> static.static(
>>> static.static("tableau_explorer/index.html")
'/bi-web-utils/daily/static/tableau_explorer/index.html'
>>> settings.FORCE_SCRIPT_NAME
'/bi-web-utils/daily/'
2

How are you running Django? uWSGI, gunicorn? (something else?)

Side note: As an issue of perspective, if you’re using FORCE_SCRIPT_NAME, it is overriding anything being supplied by the server. Also, the situation is not that the script name is being stripped by anything. The issue is that it is not being applied within the code. This implies that there’s something possibly wrong with how you’re making that setting. (Normally, I would expect to find that setting either in your settings.py or as an environment variable in the server process being read in your settings.py file.)

How are you setting this? If it’s as an environment variable, I’d verify that that is being done correctly.

3

Thanks.

I am starting django with “python manage.py runserver”

FORCE_SCRIPT_NAME is hardcoded in settings, by simply assigning a string, as below

FORCE_SCRIPT_NAME = "/bi-web-utils/daily/"

When I open a shell in the running container, and run manage.py shell, the method django.templatetags.static.static gives the desired value, that is, it starts with value of FORCE_SCRIPT_NAME. But the rendered HTML does not.

And to reiterate, sometimes when push an image with no code change, “everything works”.

DEBUG is True. I am going to add Debug Toolbar as my next step. And I will find out of the problem goes away if I run it with gunicorn.

4

Side note: This is documented as being a very bad idea. See django-admin and manage.py | Django documentation | Django
It’s most likely not the cause of the issues you’re experiencing, which is why I’ve identified this as a “side note”, but it’s still not a good idea to do this in a production environment.

For clarification - are you saying the rendered HTML does not give the desired value when you render the HTML in the shell?

If you haven’t checked that, that’s what I would look at next. Don’t just call the static function, perform the steps your view would perform to see if you can recreate the issue.
(You might want to copy a small test template into the container to make it easier to see what’s going on. I’ve found that using the render_to_string function makes things easier, too.)

5

Good suggestion, using render_to_string in shell. But so far doesn’t tell me anything… I have replicated the call, except for a valid WSGIRequest (I left “request” parameter out), and the template is generating the desired html – render_to_string result includes the FORCE_SCRIPT_NAME components in the path to .css, while the actual result of request() does not.

Another detail I left out, there are 4 pages on the site and all are effected same way: either they all get the desired path, and pages successfully load .css, or all of them get the non-FORCE_SCRIPT_NAME version of the URL, and get 404s trying to load the css.

While debugging, I did hit the debug screen and captured all the ENV variables.

Ken, or anyone, do you know if FORCE_SCRIPT_NAME is a widely used feature? I would think lots of companies have setups that involve prepending paths like this, but I don’t see much discussion of it in StackOverflow or here. It seems like whatever problem I’m hitting is probably something external to Django (or something external triggering something in Django).

6

Actually it tells me a lot. What I take from this is that fundamentally, your configuration is correct. I’d start looking beyond the basics then to see what might be causing this. Whether it’s something being cached that shouldn’t, or what, I couldn’t begin to guess. But I’d be trying to recreate the problem in an environment such that I could either connect to it with a debugger or start adding a ton of print statements to follow what’s going on.

It is for us - we use it extensively. We might deploy half a dozen django projects in uWSGI under a single nginx server, using separate paths for each project. Usually, we use the uWSGI parameter manage-script-name to ensure that paths are properly created, but we often have background (celery) tasks that run and don’t have access to the request object. So we use FORCE_SCRIPT_NAME in the celery process to ensure that rendered urls in those processes are correct.

It has never failed to work for us, and this is both for instances running natively and within docker containers. So yes, my inclination is to not believe that this issue is related to the feature itself. I’d be looking at everything other than that first - how you’re building the containers, the runtime environment being used, any middleware involved in URL rewrites, whether you’re doing something “atypical” in how you’re rendering templates, etc, etc.

That’s not so say that I couldn’t eventually come to the conclusion that the problem is within Django - but my personal experience puts that possibility near the bottom of the list.

7

Thank you, I will let you know if I get an explanation, your answers have been helpful as usual.

8

Side note: [using manage.py runserver] is documented as being a very bad idea. See django-admin and manage.py | Django documentation | Django
It’s most likely not the cause of the issues you’re experiencing, which is why I’ve identified this as a “side note”, but it’s still not a good idea to do this in a production environment.

Changing to running with gunicorn seems to have fixed the issue. It’s not a very satisfying explanation in that the sometimes-it-works-sometimes-not phenomenon isn’t explained, but I have had 15 or so deploys without recurrence of the symptom once I changed to starting django with gunicorn -b 0.0.0.0:8000 bi_web.wsgi .

So, am marking this as the solution. Thanks Ken.

9

Agreed - but I just chalk it up to being “Reason number x of y as to why you shouldn’t use runserver in a production environment.” It’s great for development work but was never built or designed for heavy-duty service. I’m sure this isn’t the only odd behavior it exhibits. So I’m satisfied with believing the docs.