Django GraphQL Websockets discussion

Using Django Getting Started
1

This is a cry-out for help maybe, or at least a conversation starter. I want to understand if I am the only one facing these issues or if this is a general issue not many people are talking about.

Context:
Often everywhere on the internet, when people talk about web frameworks, they praise Django. They say its very well done, no bugs, very solid. Now, more than a decade passed since I started developing apps and I’ve seen a lot during these years, but the state of the framework for me is in a bit weird state (will explain more in depth in a moment).

I’ve used PHP (Zend, Laravel, Codeigniter and Symfony) at the start then I’ve used the .Net frameworks and .NET MVC then some nodejs (express, koa, hapi) and some Golang (Fiber), all had differences, OOP, Functional, MVC, Code first, Configuration first, Plugins, Preprocessors but mostly I’ve been able to create anything I wanted, very smoothly.

In the past 2-3 years I’ve worked only with Python (specifically Django at work), some FastAPI project in my free time.

Now I want to create a showcase project, where I integrate some best practices, something I can showoff after so many years of experience.

And I want to choose Django, because its the framework I have a lot of experiences in my CV in the last years, and its the “de-facto” framework used in many places, and writing this app in Django maybe uppen my dev skills a bit, since I want to do a rather complex showcase app.

So my expectations were that I will mostly have “on the plate” everything I need to create anything I want, since many people say its the choice if you want a complex app, scalable and so on.

This is where the struggle starts:
I want to create a mobile app and the backend to be written in Django it will use GraphQL for obvious reasons and the app will have an internal chat that will use Websockets, I don’t want to use DRF because I’ve seen there are some performance penalties and other things with how DRF serialisers work, this will be a project, that will need to be able to scale seamlessly, so I will avoid any bloatware or something that slows me down.

  1. GraphQL:
    There is no native way to use GraphQL with Django. (What?!). Ok, there is this package graphene, and actually: django-graphene - GitHub - graphql-python/graphene-django: Build powerful, efficient, and flexible GraphQL APIs with seamless Django integration.

There is a known issue with the N+1 problem in GraphQL and the solution is to use dataloaders, but their documentation on dataloaders only has examples with async: Graphene-Python

Aparently they’ve upgraded graphene and the Promises part don’t work that well.

The issue:
They actually don’t have async views working between django-graphene/django:

There is a topic with dataloaders not working (literally they do not work and no one actually makes any effort to make them work): Dataloaders not working · Issue #1425 · graphql-python/graphene-django · GitHub

One of the key things in using GraphQL and it does not work. And the documentation does not tell you anything.

If you spin up an asgi server then you might get it to work (I couldn’t), there are other people complaining on their github, with no response (I am not sure if its actively maintained anymore).

You know right now the state of Django is, support for async is not that great and your app might be even slower using Async because of the “sync_to_async” bridge.

And people in their github say that many plugins being sync might not work with Async views and its a very big mess.

I found a workaround to use someones package and basically instead of async, use sync functions and Promises and this also avoids you needing to have a asgi server with all the bad things and told them to update at least the documentation with it, but nothing happened: Enhance docs for DataLoader & django 4.0+ async examples · Issue #1390 · graphql-python/graphene-django · GitHub.

Basically you can use a package written by someone from the community:

And this helps you write graphql dataloaders in a sync manner, without needing async server and you don’t need to add sync_to_async, async_to_sync and all the other bad things on all your methods in the app just to make it work.

But django-graphene don’t want to add this to the documentation page for some reason.

Now besides the Django Async Views are not stable, sync_to_async, using sync dataloaders instead of async… having to rely on someone’s package (maybe there are bugs) for a production app, its not really what I would describe as “Mature, Scalable, Finished” and other adjectives Django receives + performance penalties between sync_to_async (async views and such).

Plus the struggle I had to endure to actually, be able to come-up with a solution that is working?

FastAPI has a mature approach, the framework is already async, you can use the Strawberry package which is very mature, why can’t this be the case with Django?

Does anyone else had this struggle? Can you tell me your experience regarding GraphQL ? Am I doing something wrong?

  1. Websockets (Django channels and Daphne - 2 new packages I need to install, not developed by the Django team)

No native websockets support, I need to install another package Django Channels. This will again force me to move from wsgi to asgi.

As I understood, ASGI supports sync also so I guess the rest of my app will be using sync routes (with the approach above) and the Websockets will use the async part (Can someone confirm this works in production?).

This will still force me to use async functions (which I don’t have any problem doing) - but then again, I will have all the pain with sync_to_async decorators for my websocket routes. Which I don’t like from the start because this will break the consistency of my code.

Then I’ve read there is an issue with DB connections and Django (some regression from Django 4.2) that will open too many connections to the DB, and that you will need to set the CONN_MAX_AGE = 0 and use pgbouncer ?!!

This is an issue actually with the Daphne package.

There is an open issue regarding this (last post 3 months ago):
https://code.djangoproject.com/ticket/33497

Why is this issue not a priority at Django?!

And there are people here complaining that they didn’t have this issue with Django 3, but when they upgraded to Django 4.2, they prod sites started going down, because of too many connections to the DB being made!!! Seriously !!!

So for Websockets I need CONN_MAX_AGE = 0 that will affect my db performance because of new connections for each request.

Again, this I would not call, scalable, mature and so on.

I feel like, Django is in a very weird state regarding python async (with all the sync_to_async decorators), there are a lot of bugs, and you need a lot of extra packages to get stuff working, that in the end bring new bugs and performance impact.

I did not have this issue with other web frameworks in the past, FastAPI doesn’t have any of these problems, but again, I need to develop this project in Django as a showcase because this is the framework I have listen in my CV.

Did someone else had struggles with poin 2? Is there a simpler solution that is not this hackish ?

Why is everyone saying Django is very mature, scalable and so on?

But when we talk about new tech (graphql, websockets and so on) you need extra packages, decorators, perf impact, no clear documentation on the Django site on how to use them.

What is up with all this pain? Doesn’t this framework market itself as batteries included where you have everything setup for yourself and you can just use it?

Why do I feel like if I want to use new tech, I need to hack myself to it, while other frameworks from many other languages don’t have this issue and everything just works.

And the fact that I need to rely on 3rd party packages for basic stuff (looking at you django-graphene, django channels) that many times don’t reply on issues, is also really bad. Dataloaders not working when they are one of the most important things in GraphQL… why did they migrated to a new version, if they new they won’t work? Why didn’t they waited until they had a fix?

What are your opinions on this? What am I doing wrong?

2

Hey there!
Quite a long post, it will be hard to answer properly all your questions/complaints.
But i will try to.

I want to raise a question here on this sentence, specially on this:

It will use GraphQL for obvious reasons

This “obvious reasons” may be obvious to you, but this doesn’t mean that they’re common sense. On my company we have done a lot of mobile apps using GraphQL (with nodeJs) and REST (with nodeJs and Django) and we have a lot of problems regarding the infrastructure of the code as the product evolves along the years, we can’t change any bits of how it’s implemented because it won’t work without it. GraphQL, on my opinion, tie yourself up with it, and everything is done in order for this integration “magic” to work. We didn’t have this issues while working with REST, and didn’t have any big performance issues on other mobile apps that used REST instead of GraphQL.

Moving on…

Let’s be honest, async is a pretty new topic in python (~5 years) comparing it to how long Django has been around (~20 years). Django was built when there was no “async”, there were only “wsgi” and no “asgi” at the time. Django, and when i say Django i mean the DSF (Django Software Foundation) and not 3rd party libraries from the community of Django, is progressively adding Async support (ASGI), and this is very well stated on the first lines of the documentation on Async Support:

Django has support for writing asynchronous (“async”) views, along with an entirely async-enabled request stack if you are running under ASGI. Async views will still work under WSGI, but with performance penalties, and without the ability to have efficient long-running requests.
We’re still working on async support for the ORM and other parts of Django. You can expect to see this in future releases. For now, you can use the sync_to_async() adapter to interact with the sync parts of Django.

This is not hidden from you, it’s really well stated on the documentation.

FastAPI is very mature on the async side because, again, it was born after the async scenario has been created on python, so it was designed with async on mind. But with GraphQL you have the same situation with Django, you need to install a 3rd party library for it to work. Its also a good thing to remind that Django is releasing in the next few week it’s 5th major version of the library, so you have years of development of having backwards compatibility and security patches, and FastAPI it’s still on 0.96 ish i think. I don’t want to remove the merit of Tiangolo or FastAPI, i really like the developer and the project, but it’s not fair to make a comparision between the frameworks/projects, because they are really different.
FastAPI was designed for you to create REST APIs (you can also serve templates using a thirdy party library) but Django was designed for you to create Web applications, but you can also use it for several other use cases, like REST APIs and others. That comes built-in with Django, external packages gives you a set of abstractions on top of that, the same you would do it in other frameworks.

Moving on…

Because it is, on what it was designed to do.

The web has changed a lot in the last 5-10 years, and we several new stuff that we can use, but not everything that’s new is essentially better than whats older, that’s why you see a lot of engagement on the community for htmx instead of all the bloat of JS frameworks out there.
It’s up to you to decide which technologies you’re going to apply on your project, and decide if Django (or any other language, framework) is the best option to use on that scenario, and sometimes the answer is just: no.

I hope that this bring you some clarity, but keep in mind that all of the statements here do not reflect the opinion of DSF or Django itself, they’re my opinion and my POV.
Cheers!

1 Like