Already asked at SO: I’ve followed @pauloxnet 's great tutorial but keep running out of ram / the browser crashes, presumably because my DB has millions of entries and I don’t think anything’s limiting the number of results.
How can we limit the number of responses, if that’d be the issue? Someone responded to my question recommending Pagination, and although I implemented the solution without error this now doesn’t show any results. Is this the right way to go about it? Is there anything I can do to elaborate on InBBoxFilter? And for that matter, is there any way to examine the actual query run in Postgres?
I’m new to everything so don’t know how to best troubleshoot this, so please forgive me asking such a stupid question. Also, I don’t yet understand django-rest-framework, so if this is a DRF specific issue please feel free to redirect me to wherever I should ask such questions.
Thank you for any help!
1 Like
Hi Chris,
There are a couple of options for getting the raw SQL. Some are easier in a shell, others you can drop into the code in your application.
print(queryset.query)- Print the connection’s queries:
from django.db import connection
# eval the queryset such as list(queryset)
print(connection.queries)
- Use the Django Debug Toolbar
- You can also use
queryset.explain() to use the underlying database’s EXPLAIN feature to see what’s going on within the query itself too.
How can we limit the number of responses, if that’d be the issue?
What would you like to limit the responses to?
1 Like
Tim, thank you very much for the reply! I hadn’t enabled email notifications and I didn’t think to check this thread until now.
In the SO thread someone advised using GeoJsonPagination, which greatly helped. I’ve since realized browsers still process markers outside of Leaflet’s bbox, and asked another question on how to stop this, although that’s no longer a Django specific issue. Might as well ask: do you happen to know if there’s a way to elaborate on .addTo(map) to only add the marker if it’s not already there? Or how to drop markers users should no longer be concerned for? Perhaps there’s a more elegant Django solution for this?
I’m already used Django Debug Toolbar with another template which has been amazing showing how queries were run, but it hasn’t been fully working with this new map template. I think it might be to do with running the map through serializers.py, viewsets.py, api.py, and new functionality I don’t yet well understand. Any idea how I could troubleshoot?
Also, I don’t know where I might be able to add your suggested print(queryset.query). I’ve used similar in views.py but could I place this in viewsets.py somehow? I’ve tried placing it in MarkersViewSet to seemingly no effect.
Thank you so much for the help!
I’m not sure if this is really along the lines of what you’re looking for, but I can tell you what we do.
You see a lot of examples of something like: L.marker(lat, lon).addTo(map). This creates an instance of a marker object and adds it to the map, but effectively ignores the reference to that marker.
What we do is track all markers in a dict (actually, a JavaScript object), and can then selectively add or remove them from a map.
Example:
points = {}'
points[point_id] = L.marker(point_lat, point_lon);
...
points[point_id].addTo(map);
...
points[point_id].remove();
...
However, depending upon the number of points involved, I’m not sure that you’re going to be that much more efficient with handling the display / removal of points from the map than Leaflet is going to be, unless you have a way of organizing your points such that they can be filtered quickly - at least to get “close” to what you need. (What I’d actually recommend in this case is that your “pre-filter” be a bit larger than the actual map bounding box to ensure no points are missed - and then let leaflet do the final decisions from that set.)
Handling zoom and scroll events might be a bit tricky as well. If you remove all points and then re-add them, it’s going to add a “flicker” to the display that may not be considered desirable. You might want to keep a separate index of points being displayed and only remove those points no longer of use.
From what I can tell, it all can be done. (We’ve never had to go that far, I’m only dealing with a couple thousand points.)
2 Likes
Your solutions sound great but I’m new to programming and especially JS so I’m unsure how to approach implementing that. Complicating this, my DB has millions of entries so a solution which drops markers outside of the bbox might be best. Do you know of any working examples I could follow?
I think there are two aspects to the issue:
- Leaflet continues processing markers outside of users’ interest
- duplicates of the same entry might be added (although unsure how to confirm that)
Come to think of it, I don’t understand why @pauloxnet’s tutorial loads the data asynchronously and adds new markers to previous responses instead of simply replacing markers with markers from new responses. Does anyone know of an easy way to do that? Is there some benefit to adding markers I haven’t considered?
I’ve been totally stuck here banging my head against the wall for a week now so thank you all sincerely for the help!
That implies (to me, anyway), that every time the user pans or zooms the map that you’re going to query the database for the points to be included. I don’t have any direct examples, but I do know that PostGIS does allow you to query by a bounding box.
I’m going to guess (and that’s all this is, a guess) that Leaflet is written to prevent creating multiple markers at the same location. Therefore, adding all the new markers is only going to add the new markers exposed in the view. So if you’re only moving the map a little bit, you’re (relatively speaking) adding just the new points and not adding all the other points already on the map.
He’s also not tracking the individual points being added. (This is the issue I addressed in my previous response.)
No, managing a set of displayed markers adds (IMO) quite a bit of complexity to the implementation.
Yes, it’s simpler and more direct to effectively ignore the markers added previously.
Keep in mind, you’ve mostly been reading tutorials at this point. These are code fragments designed and intended to get you started quickly. A tutorial, by necessity, must seriously limit the scope of what’s being taught.
More significant “point management” is going to be a more detailed and intricate issue, which is probably of more limited interest to the general user-base.
Yeah, I think you’re right. I have no interest in writing custom advanced code for arcane functions but efficiently managing these millions of entries is unfortunately the basis of my project 
There can be multiple entries corresponding to the same location and Leaflet definitely doesn’t prevent this. I’m uncertain whether it prevents duplicate entries, although marker shadows darkening indicating multiple markers strongly suggests duplicates aren’t prevented.
Anyway, these questions relate more to Leaflet and JS than Django and I don’t want to waste the time of people here. Does anyone know of a good forum where I can ask such Leaflet / JS questions? Or know of any more examples / tutorials I could follow? @pauloxnet, can I persuade you into making more tutorials? 
Righto, I asked over at the Leaflet GitHub and was summarily shut down. This guy doesn’t owe me anything and I’m new to GitHub etiquette but is closing issues on people like this discourteous?
Anyone know somewhere beginners can ask dumbass JS questions? Already tried StackOverflow and have no idea how to “track duplicate markers with a JS Map object with marker location (or id if provided by server) as key, and marker object as value” .
He’s basically suggesting what I mentioned that we do in our project (How to limit entries or improve performance with GeoDjango map? - #4 by KenWhitesell)
This really isn’t Leaflet-specific, and to a great degree not even JavaScript-specific. The implementation may be in JavaScript, but the algorithms involved wouldn’t be.
Briefly, you can create one or more objects to help you tract the current state of your system. You would have a points object as I described above. You might even create it with multiple attributes, where the marker object is just one of them. (The lat, lon, and “shown on map” values might be others.)
You possibly may want another set of nested objects to create an index of these points (e.g. a structure where point_idx[lat][lon] == point_id (or point_ids)). This is to help you identify which points fit within a bounding box.
The bottom line seems to be that you’re going to need to do all this data management yourself. Leaflet doesn’t appear to do any of it for you. When you read the data from the server, you’re going to need to pre-process it into these objects.
However, what I can’t determine is whether or not even that’s going to be enough. It may end up being the case that JavaScript just can’t reasonably manage 1,000,000+ points. (I’m just intrigued enough by the issue that I’m going to give a quick try to see what happens if I add 1,000,000 points to a map.)
So just to follow-up on this, some quick ad-hoc testing shows that Leaflet effectively fails somewhere short of 10,000 markers on the map. (In testing on my system, it becomes effectively unusable at about 4,000 markers.)
See the Performance section of the FAQ: Leaflet/FAQ.md at master · Leaflet/Leaflet · GitHub
I kinda doubt you’re going to be able to do anything to effectively manage 1,000,000 points.
Forgot to update that I’ve since resolved this through better understanding what I needed.
Markers of previous responses are irrelevant so it’s easiest to simply drop them when a new response is received. Sorry for asking such an unnecessarily complicated question!