What is the most efficient way to store context data?

Using Django
1

I am building an app which goal is to perform some calculations based on user input that user enter in a form.

In order to respect DRY approach, I have created a Class which contains some data that I use everywhere in the site (template views, form view, and the core python calculation script).

For the html templates, I access the data using get_context_data() in my views. It works well. Especially for the template views which are static so I can cache them and have blazing fast performance.

However, I cannot cache my main formview since it’s dynamic. I noticed that the method get_context_data() is really slow and takes about 80% of my render time. So I think my approach is not the best one.

SimulateurData Class is stored in a python file called simulateur_data.py and contains the following code:

from django.db import models

    class SimulateurData(models.Model):

        def __str__(self):
            return 'Simulateur Data'

        data_value_1 = XXX
        data_value_2 = f(data_value_1)
        etc ...

As you can see it’s not really a model since it’s not saved in a models.py of an installed app. SimulateurData also contains some calculations since some values inside are computed from other values. Since these data never change, I do not want django to compute them each time. What would be the best approach to create, store, and reused SimulateurData as context data for the whole app, templates, views and python scripts included?

2

I can only make some guesses here because you didn’t really include enough details for me to be sure, but I’ll offer some thoughts on this.

  • Put this data into a table.
    • Perform the calculations once (perhaps using a management command) and store the calculated results in the table
  • Retrieve what you need when you need it
    • Do you really need every value on every page?
    • Is this data combined (calculated with) other data, or just rendered as-is on each page?
      • If some of the data is just rendered on the page, you could pre-render that component and store it in cache.
  • Make sure that you’re really solving the right problem.
    • 80% of the render time may be immaterial if the entire page renders “fast enough”.
    • A performance issue may not be where you think it is… Be very careful about drawing conclusions regarding the location of those issues.
1 Like
3

Thank you Ken for your kind answer and insights. Here some are some questions and comments I have regarding what you said:

  • Put this data into a table.
  • Perform the calculations once (perhaps using a management command) and store the calculated results in the table

What is the proper way to do it (put in table and then perform the calculations once)?

  • Do you really need every value on every page?

No, I only need some of the data on some pages. My goal was to have all the data I need for the whole web site in a single file, so it’s easier to maintain. As I said, I need this data:

  • to pass it to html static pages as content information (that’s why I thought using get_context_data() was the proper way to solve this problem)

  • to perform calculations from my main python script, via import SimulateurData python command

The data contained in SimulateurData is independent from user and not updated often. I would like the calculations to be performed only once and then store the results to use them directly (for example, perform the calculations each time python manage.py runserver command is run, then store the results and use them when needed).

  • Is this data combined (calculated with) other data, or just rendered as-is on each page?

No, the data is not combined with other data. All the calculations performed into SimulateurData use data from SimulateurData.

  • Make sure that you’re really solving the right problem.
  • 80% of the render time may be immaterial if the entire page renders “fast enough”.
  • A performance issue may not be where you think it is… Be very careful about drawing conclusions regarding the location of those issues.

Do you have suggestions about how I can track the performance issue more accurately and then be sure that I am solving the right problem?

4

Continuing the conversation…

Create your table(s) to hold the data, then write a management command that uses your data class and runs your calculations, and then stores the data (and calculation results) in the table. Run that command once and it’s done.

You could also create your data as a fixture and use the loaddata command to load it into the table.

Or, you could also write a set of SQL Insert statements to load the data directly.

Use whatever you’re most comfortable with - the specific method doesn’t matter much.

Since this is static data, and you’re using it for static pages in some cases, you probably could cache those entire pages.

For the non-static pages, look to see if you can cache the rendered components based on your static data.

Django Debug Toolbar (DDT) is an easy first step. It’ll make a lot of information available to you for further investigation. (Everything after that is going to depend upon what you find out here.

You didn’t mention how you came up with that 80% number - if you’re already using DDT, then there’s a lot more information available beyond that percentage.

But one of the general principles of Application Performance Management is knowing what your goals and objectives are. To simply say you want your application to respond “as quickly as possible” is a bad choice. Sure, getting page response from 3 seconds to 1 second can be extremely important in some circles. However, getting page response from .3 seconds to .1 second is rarely worthwhile, and I would suggest that getting page response from 30 ms to 10ms is never worth the effort.
So once you’ve set your desired standard, then you can look at what you can do to achieve it.

1 Like
5

Ken, first I would like to thank you for your time and kindness. I really appreciate your helpful and constructive answers. I will look and investigate further regarding the first point (data into table and management commands).

For the second point, I have just implemented fragment caching in the dynamic page for all the context_data fragments. Just this point has already greatly improved the rendering time. The get_context_data() function in my views.py dropped from 0.3 second to 0, meaning that I use cache instead of reading each time from SimulateurData. For example in my dynamic pages html templates:

{% load cache %}
{% cache 600000 fragment_cache_name %}
<p>{{data.data_variable_name}}</p>
{% endcache %}

I was already caching entirely the static pages (directly implemented in this case in my urls.py file)

Regarding the third point, I fully agree with you regarding performance, it’s a question of compromise. I am in general trying to find the good balance between speed and efficiency/maintanability. Not always easy but it also helps discovering new methods and have a better understanding of Django in general. I will take a look at django debug toolbar to improve my understanding of what’s going on.

Have a nice day Ken!
Cheers from France

1 Like