I have a GPS tracking app built with Django. Right now, it works for a few devices, but I want it to handle thousands of devices sending location updates at the same time. How do I make my system faster, more reliable, and capable of handling so many users without crashing?
There are no “Do this and your problems are solved” answers to questions like this.
The first step is to understand the flow of the data. Are these updates consistent and regular, or are they intermittent and periodic? Do they happen for only part of the day, or continuously over a full 24 hour day?
Where are the bottlenecks? What testing have you done to identify where things are failing?
Django isn’t the only component involved here - you will also have (at least) a webserver and a database involved, each with their own set of constraints.
You also need to understand how this data is being used after it has been collected - this may also affect your solution.
For example, I’ve worked on two different projects. In one project, we were receiving up to 15 million rows of data per day. But the rate of submissions was highly variable. There would be two times a day when the traffic would peak to a rate of 5 to 6 million rows per hour, and would fall off to near zero from 10 pm - 5 am. Because of this, we could queue the data for processing, allowing the system to “catch up” at night for data that hadn’t been updated during the day.
The other project had to accept 10 rows per device per second, every second. This means that with 80 devices, we were getting a consistent 800 rows per second, or nearly 3 million rows per hour. But there was no “down time” involved. We couldn’t allow the ingestion process to fall behind - there was no way to catch up if you couldn’t keep up.
The solutions to both of these situations was different. But the key point here is that you need to evaluate this as a complete system, and not just look at Django.