I am a self-taught developer and django user, building a solution for my company.
The project will provide a facility to perform searches across company datasets for multiple values at one time.
- Each value is identified by data type (e.g. phone numbers, email addresses, mac addresses, IMEI numbers)
- Dependant on the data type different tables are searched for matches (all django) and type-specific functions are run. (external API calls, lookups, string manipulation etc)
- Search values and relevant results are displayed row by row in a table
Crucially none of the searched values ever hit the database, this is all a runtime.
My approach so far has been to do this via custom classes and methods which I’ve built in managers.py and call from views.py, but the more I consider the issue and learn about the framework the more I question this.
I’m also detecting that some of my class methods are being called elsewhere in the framework ( _resolve_lookup for one) which can’t be efficient?
How would you suggest I approach the problem?
Should I be defining each of these data types and custom functions in models.py with managed = False?
Would I benefit somehow by putting the searched values into the database?
full disclosure I already asked this question on Reddit but didn’t find a solution other than “do the tutorial again”
TIA
You’d definitely benefit from putting them into a database - databases are built literally to do searches across datasets, and they’re really quite good at it.
Django’s model system is designed to pull from a database, so you won’t be able to use it on data that’s in custom classes and methods; if you keep going down that road, you will have to keep writing all the searching logic yourself.
One thing you could look into if the data is relatively static is to use an SQLite database - they’re designed primarily for reading, not writing, but it could be a good mid-step if you’re going from everything in memory, and it doesn’t need a separate process running (and Django has great support for it).
Thanks, Andrew,
Apologies, the data being searched is 100% in databases all mapped by models - I like a challenge but reinventing the wheel is something else!
It’s the query values that don’t go near the DB and its the code to process those values, defining/creating objects for each and calling APIs that my question hinges on.
for example;
user inputs [1] 0121 876 5437, [2] 07967 456321 and [3] 356084096345678 along with some metadata about the search, all handled via a django form
the custom code strips whitespace out and runs regex to identify [1] as a UK landline, [2] as a mobile phone and [3] as an IMEI
searches are made on the django managed databases relevant for [1,2 & 3]
custom code runs on [1] to i) extract area code portion of number ii) perform Google search API call
and on [3] to make API call on IMEI manufacturer database API
the results of the database search and custom code gets fed back to the user via a view
Hope that explains better
thanks
Ah, OK, I misinterpreted what you meant by “non-database data” - my mistake. Traditionally, how I handle input data like that is via the Form class - that way, on input, it is sanitised and turned into a searchable value as the form is validated.
Honestly, though, this kind of code just has to go somewhere. Form or model is a fine place for it, but I’d just do it as a custom method rather than a whole class; I presume your end result with the input data is a set of queryset filters? You could have a method that outputs a queryset or Q() object, maybe.
haha I can see why… I guess my specifics were about the non db data and the detail surrounding it is the db data!
Anyway, wanted to come back to say thanks… I revisited my models and once I got started with a few custom functions it all started to fall into place
Thanks for the help