As part of a project using Django, I have 409 static data pairs I want to load on startup and make available in memory to all user sessions .
My problem’s not knowing what this technique or method is called so I can do the research to implement it. I don’t want a handout, just a pointer in the right direction.
Thank you all much in advance.
See the Django Cache Framework, particularly the section on The Low-level cache API. You can store data there.
You can also create your own cache outside of Django’s facilities using Redis or Memcached.
However, unless you have really unusual requirements, I wouldn’t bother. I’d store the data in the database and grab the data when needed. If the data is used frequently enough, it’s going to remain cached by the database and so the query to retrieve it is not going to cause physical disk IO.
(Keep in mind that by default, Django uses the database for session data. You’re not saving anything by preloading data into the session from the database.)
Also be aware that in any production-like environment, your Django application is running in multiple instances - and possibly across multiple physical systems. What’s “in memory” in one instance isn’t necessarily in memory in a different instance.
So, my bottom line is, put the data in the database and access it when necessary. If you ever find that it becomes a bottleneck in your application, then (and only then) look for a way to optimize that access.
Ken,
Thank you for the concise information. The question was a bit too open ended, I’ll admit. But it served the purpose.
I’m quite ignorant on the programming side of the house in IT. Flip side, I’ve worked all the other areas since '88, from the desktop hardware to enterprise eco-systems management in the Fortune 20, LAN/WAN/SANS ect.
I’m aware or the tradeoffs and limitations you pointed out and dealt with them more times I can count from projects pushed down from the top and ignorance on the bottom side.
Best to view me as a novice who ‘just knows’ he’s working on the next ‘big’ thing and wants the project built from that premise and humor me as deluded.
I can say this much … Nary a single instance available on the web. Period. Not available to the avg laymen at all. Licensed only in the largest companies and Government agencies with ‘per seat’ licenses on the MS Windows operating system.
Patents expired on it in 2017. That year I began work on this project to convert it to laymens terms and present it in a manner that didn’t require years of learning to master.
None of this matters to the community. I have to build up my chops, build a reputation solid enough to ask for, and get volunteers to help. And that isn’t going to happen overnight.
I appreciate the background. Now, from the flip side, I’ve been programming professionally since '78 - I learned how to write code using an IBM-029 keypunch system. I can’t count the number of different hardware and software platforms that I’ve had my fingers in over the years.
My career path has taken me through three different supervisory / managerial positions along with stints as an Information Security Officer and as an Enterprise Systems Architect. (Fancy titles for jobs consisting mostly of pushing paper.)
There are some constants that I’ve observed universally over that time, and one of them is for some programmers to develop a preoccupation with “micro-optimizations”. If you’ve got a sub-optimal algorithm, that’s going to have significantly more effect on overall throughput than worrying about minor details. Saving 250 milliseconds on a request taking 5000 is just not going to create a noticeable improvement.
Bottom line is that any effort to optimize performance at this stage is completely misplaced. You don’t know where your bottlenecks are going to be until you have a system up and running and can actually measure the performance of your system.
You’ve got time. There is nothing you’re going to be able to tell me that will convince me otherwise.
Get your system up and running. Build your test stack to see how it will perform under load. Then measure the performance of the different components within your system, identify the bottlenecks and spend your time addressing them.
That’s 43 years of directly applicable experience trying to explain why your efforts and attention to that topic is, at this time, misplaced.
With that context, you are entirely correct. That is why I don’t pretend I know, or even would begin know the answers on the programming side. I would rather ask than sit in silence. Stupid is incurable, but ignorance is simply the lack of knowledge.
In hindsight, it was a bit silly since the static code represents an extremely small footprint even with or without an optimized platform. The static data is unlikely to grow more that a few lines in the foreseeable future, and I should seen it as a non-issue.
Thank you for your explanation and top down view. Your patience while I fumbled around seeking answers is much appreciated.
No worries - I’m here to teach. Feel free to come back with any more questions or issues you may have.