Help with project structure, sharing models across apps?

Using Django
1

I have 5 APIs (and growing) that regularly download (currently outside Django, but will be migrating in) into seperate database tables, roughly one per API with relationships. Machine sensor data (x2), weather, geological, geocoding + whatever else comes up in the future. The relationships are mostly based on per machine and per sensor type, with the “master” being per machine. Also within those API apps, there are a few functions that process the raw sensor data (a lot of it, and messy) into other tables, e.g. cleaned, first/last/average values for fast user display on dashboards per machine.
Raw data is used for stat analysis and ML prediction models.

My thinking at the moment is to create a polling app which would host (not sure of the right term) those tables (models). These apps would manage the data in those tables. Apart from users, there is little or no user Create, Update or Delete interaction with the database data. Mostly Read.

I would then create other apps that would bring together all these various tables for display to different types of users in dashboards. However this would mean importing the polling app models into the other apps, and everything I have read says that all apps should be self contained. Also there would be relationships across apps which makes me shudder. Again all should be coming from the machine id, not the other way round so trying to avoid circular dependencies.

The other apps will be things like machine summary, fleet summary, aftermarket sales and technical support dashboards which will use the same models (albeit with slightly different queries), but feel like different apps. There will be some new models for the different apps, however they will share the same core processed API data.

For various reasons, I already have a proof of concept built in MATLAB with the database tables up and running and works…as a PoC. My thinking of the structure probably comes from creating that.

I am now learning Django for the production version, however I am struggling to see how to fit within the principles of Django I have picked up:

  • Keep apps small
  • Each app does one thing well
  • Keep apps separate so they can be exported and used elsewhere

Is this structure normal/sensible for Django, and apart from creating one massive app, should I be rethinking the approach? Help!

2

Unfortunaley I don’t have a solution to your problem. However we’re facing a similar problem so maybe our experience can help a little.

What we ended up doing breaks the principles you cited. The code is still seperated into apps by concern/responsbility of the code, but dependencies reach across apps and removing one app would break the whole project. Maybe it is possible to work on a principle where each dependency on another app should have a fallback in case it can’t be resolved. However for some things I just don’t see what a suitable fallback would be.

Another thing we had some problems with was circular dependencies, which have the annoying habit of not always being obvious. This is not Django specific, but it can happen. When it does this is an indicator that the conflicting code should be moved somewhere else in the hierarchy. This has become less of a problem over time since we learned to seperate code along the guidelines Django provides. Maybe it seems silly that a small helper function gets its own file when all forms of an app are smushed into the same one, but this does help with organization.

That syndrome of bloated files is also a concern to be wary of. Fortunately we only had to split files containing tests so far, but at some point splitting e.g. a forms file into several may be a good idea too.

3

In theory, it’s possible to create completely isolated apps, but then you’d have to figure out a way how to make these apps talk to each other. For example, every app could expose a rest API endpoint, and other apps could use those endpoints to communicate with each other. Everything is possible but at a price of complexity.

All in all, the app term in Django is a bit misleading. I suggest thinking about apps as you’d thought about modules or plugins. Use them to split logic by domain or any other way you feel suitable.

Also, think about implementing two types of apps: 1) independent lego bricks that you could reuse in other projects and 2) orchestrator apps, that glue lego bricks together and use them to implement the main business logic

4

Thanks for your experience in this area. I think you are right that there is no way of completely splitting the apps as there are just too many interdependencies. I am hoping that apps will be a good way of gradually adding functionality and will look into how best to split the business logic.

5

Timonweb, I like analogy of the lego brincks that I could take to other apps and the glue of the orchestrator apps. Will have a think about how best to define the interfaces between all these apps. APIs do feel like an over complication at the moment but you never know! I do have to create an API as there is an external webapp that needs the results of the prediction models but much simpler than what it would take for the intercommunication. At least the current thinking is not completely out there!

1 Like
6

There could be various solutions for solving your problem, however, once the article about salesforce communities implementation helped me greatly when I was faced with a similar problem