Structuring large/complex Django projects, and using a services layer in Django projects

Using Django Getting Started
22

Hello everyone,

I’m sorry for bringing this topic back to life, just wanted to add some context.

First - there are a lot of great ideas & suggestions here. Kudos :raised_hands:

The important takeaway seems to be - people should use whatever makes sense to them & their teams.

As for the mentioned styleguide - this is what ended up working for us, for the past 8 years of developing Django projects. And it works on a big scale projects.

Sometimes, the service layer is not always looking like a single file with a bunch of functions, but rather like an entire Python module with well-defined interface, so we can encapsulate something complex & isolate it from the rest of the app.

We’ve even gone further creating a layer of types, between the ORM & the business logic (this is an overkill for most cases, but there are cases where it helps)

The general idea - separation of concerns - is key.

Additionally, we are constantly adding (well, at least for the past year) new things to the styleguide, as we learn them & see them in action.

There’s still a lot to add to the styleguide, of course, but we are getting there.

Right now, we are running a survey, to gather some feedback and know what to improve.

Again, kudos for the valuable feedback & points of view.

BTW, if someone wants to discuss the related ideas, I’m more than happy to do so. Perhaps it’d be best to do so on some kind of a call.

Cheers!

1 Like
23

Hi James,

I appreciate your pragmatic Youtube videos/articles,
Have you seen this video on clean architecture in Django:

I think the speaker agress with you when you say

To really do a backend-agnostic service layer on top of the Django ORM, you need to replace or abstract away some of its most fundamental and convenient abstractions.

And he did just that and made Djaq to replace the ORM and decouple the ORM from the View.

1 Like
24

Great thread! Thanks for all the information and also for the articles by ubernostrum.

To me it feels that moving code that fits into the model, QuerySet, or Manager certainly makes tons of sense. This does not contradict having a “service layer”, though. The question where to put complex queries involving 3rd party calls, and logic that deals with multiple models was still not answered by ubernostrum, sorry.

I have read the follow-up article and it basically says “well, use appropriate patterns, there is no one-fits-all solution, but service layer is not solving it”. Sure. But the idea of the “service layer”, or however you call it, is to separate logic that does NOT belong into:

  • views
  • serialisers (if DRF used)
  • model
  • QuerySet
  • Manager

So, it makes perfect sense for code that deals with 3rd party calls or multiple models. You can call it differently or have some patterns around it, so that you don’t have outsourced code within a file services.py (not sure anyone would propose this?) or a package BUT it would logically still belong into a different layer. And that seems exactly right.

A layer may contain patterns. A layer does not mean that it is one file with a huge function. But it will still lead to a separation of concerns, which helps for code reusability, ease of testing, and having model/view clear of external dependencies.

So

It doesn’t make sense to do all those things in a method on a single model. But the conclusion — you need a service layer for this — doesn’t follow. First of all, hoisting those half-dozen (or potentially more) things out of a single method on a BillingAgreement model and into a single method on, say, a BillingService doesn’t actually solve the real problem, which is that a single method is trying to do too much.

Sure, this would be horrible. But for me this is not what “service layer” entails at all.

So maybe it is just not clear enough what service layer means. Usually, you have a domain layer and application layer (these are sometimes mixed but in any case clear of API/views and clear of ORM). There is a reason why the “enterprise guys” in C#/Java/… separate application from database and these languages also have great ORMs and STILL went with separating it.

I also like GitHub - HackSoftware/Django-Styleguide: Django styleguide used in HackSoft projects because it went through some refinement and discussions that were integrated there. The paragraph on the question “Why not put your business logic in custom managers and/or querysets?” puts it nicely.

A hybrid mix could also make sense to me: Clearly use Managers, QuerySets, etc. but keep views lean. I also don’t get why “web app” is sometimes mentioned. This whole thread is about enterprise applications. View is a big part of it, but most of functionality should not depend on the UI that uses it.

Would love to see this discussion going on! I hope I didn’t step on too many feet, toes, or other body parts here.

2 Likes
26

I posted this thread a little while ago now, but this still something I regularly ponder. As my product and team grows in size, it’s becoming less and less of a hypothetical / thought experiment, and more and more of an actual problem that necessitates attention.

I’m still playing in hypothetical land, and I admittedly haven’t gone back and re-read this whole thread immediately before writing this, so I may be misrepresenting some viewpoints. Forgive me, this has turned into a big thread! If I have done so, I apologise. It is not done in malice. I’m not picking sides. I am (still) not sure enough of myself to do that!

Anyway!

My main hypothetical issue / concern with a hybrid approach is that it does not address a key desire of mine, which is a consistent developer expectation when working on a project.

The appeal to me of a strict services layer is that each routine is an intentional and considered means of interaction with the system. Which I guess might be obvious. It’s very…DDD.

Django does a whole lot for you. That’s its intention. I am sure that we are all on board with this as a benefit—at least in some or even a lot of situations—else we’d probably be using something else. The flip side of this is that when you, for example, create a new model, you need to consider all the entry points that Django is going to provide, and whether they are all going to take into account your domain / business logic.

I personally do not believe that Django’s architecture exactly makes this a cake walk. We could talk all day about whether or not I am right or wrong. In fact, there are plenty of people, including ones I deeply respect the professional competencies of, including people in this thread, who seemingly disagree with me here. I’d be the first to say that this belief could very well be me not knowing enough about Django. However as someone that’s been working with Django full-time for nearly a decade now, I’d argue that this is at the very least not easy or intuitive to do.

The reality of Django, as with all frameworks, is that making assumptions is necessary to provide the abstractions that it does. This means that certain hooks / entrypoints which Django intends for you to insert your unique business logic may not be suitable / capable of allowing this in your unique situation. This means that more complex Django projects may either override more and more of Django’s batteries (which introduces its own complexity which has to be argued for), or may instead introduce rules / conventions—whether implicit or explicit—pertaining to which Django ‘batteries’ you should and should not use.

My issue with a ‘hybrid’ approach (in the way that I’ve interpreted it, which again could be wrong) is that it would necessitate some complex conventions / rules around how you should accomplish various things in various circumstances. This makes it harder for new developers to jump into a project with the confidence to get stuff done. Hell, even someone deeply familiar with the project could forget gotcha A or footgun B. Is it easier to say “only use ModelForm in these circumstances?”, or simply “we do not use ModelForm”?. I’d say the latter. A big thing for me here is model validation. If you can’t get Django model validation to do what you want in all circumstances, then the contract is broken, and this is something that needs to be communicated. This isn’t a unique fault of Django’s by any stretch. But the reality is that it’s something that needs to be considered in a project.

We could also talk all day about why Django is this way. Like all things i believe that it is a combination of Django’s origins as essentially publishing software (though it is far far more than that now), ensuring backwards compatibility, required effort, and that addressing this enterprise-level architectures is not as high a priority for the largely-volunteers that work on Django. Which is fine!

As I said up front, I’m not sure enough of myself to want to pick a side. However when it comes time to start more purposely addressing this at my day job, I’m erring on the side of a more comprehensive services layer.

1 Like