Seeking advice for WebGIS components

Hello.

I am new to Django and building my first non-test site.

The main feature is supposed to be a WebGIS map that the users may use for searching places. It must not be collaborative (users don’t edit it). There will be only a few layers (probably just one, maybe 2 or 3, plus the OpenStreetMap background layer).

In addition to that, I need some functionalities that Django easily offers, in particular user authentication and permissions control so Django seemed a good choice.

I have installed the GeoDjango apps (so, OpenLayers) as well as Django-Leaflet and gave them both a try in the admin area of the web site. It works superbly.

However, if in the admin area, OpenLayers or Leaflet only need to display one GIS feature at the time, it is very different in the user pages. My main map (and also my main Django app) will need to display many, many GIS features (several hundreds of thousands of points on a country map).
I tried serving directly the content to Leaflet by AJAX following this example from the author of Django-leaflet by serializing the features as GeoJSON but the performances were just not acceptable : it takes several minutes to render the map (and again each time the user moves the map or zoom in or out). I guess it’s simply too heavy. I should have even more GIS features in the future and it will probably get worse.

In addition, I have other requirements :

  • The users must be able to filter the GIS features that are displayed on the map according to several criteria. Basically, it should behave like the maps on AirBnb or Booking.com, where you can refine your search to filter only accommodations with Wifi, attached bathroom, by price range, etc …

  • They need to be able to open another web page for each GIS feature (this should be easy with Leaflet popups.

  • I need strong user control : verifying authentication of users seeing the map, verifying their permissions and logging their use of the map.

My database is Postgresql + Postgis, as you could expect. The data to be displayed on a map has been prepared a Postgresql MATERIALIZED VIEW that has indexes on all the columns that may be involved in the user’s searches. Direct SQL tests on the command line showed good response times (less than 0,5 second for a query with several filtering criteria).

I kind of understood that I now need to set up a WMS server (not a WFS one, I guess), but I don’t know which one could match the requirements mentioned above and how to integrate it with Django.

Hopefully, some of you guys will have some hint …

My experience with Leaflet and GeoJson is much more limited - I’m only displaying tens of points at any one time.

Some general thoughts -

  • The article you referenced (GeoDjango maps with Leaflet) mentions that “A map with more than 12 000 HTML objects is not going to be snappy.” I have no idea what that translates to in terms of performance on your site.
    You might want to do some testing to determine at what quantity of points the performance stops being acceptable. You could probably try this most easily by adding a limit to your queryset in your view that the AJAX call is using to retrieve data.

  • If you’re working with point data and not areas, polygons, etc, GeoJSON seems to me to be “heavier” than you need. There’s probably a lot of overhead that could be avoided by simply sending the points as a simpler JSON structure. (e.g. list of lists of coordinates).

  • Make sure your code is structured such that you’re either only

    • retrieving the data once and not every time the map is scrolled or scaled
      or
    • only retrieving the points for the current viewport.
  • Your other requirements are no problem. Django can handle them easily, assuming you have the data to support it.

  • We don’t use either WMS or WFS with Leaflet. We use a standard Apache server with mod_tile and mapnik for rendering. (We host our own tile server with rendered OSM data and a custom stylesheet for handling zoom levels up to 25.)

  • Opinion - I’m not sure just what value you might get out of trying to individually render > 100,000 points on any one map. You might want to look at some of the options for clustering or even heat maps

Ken

Hi Ken. Thanks for the tips.
When you’re writing :

Do you mean the Leaflet code or the Django code ?

I’m referring to your JavaScript code that is making the AJAX calls.

When you’re addressing performance-related issues, the first step is always to figure out where the time is being spent. You mentioned that it takes “minutes” to render the map. Your first objective then is to figure out where those minutes are being spent, and attack that piece vigorously.

Hello Ken.

I am doing some tests and I believe that using a WMS server is the way to go :

  • Using GeoJSON served through Ajax, Leaflet was reasonably fluid up to 1000 features. At 10.000 features it was already too slow. So even with improvements such as heat maps or using JSON, I don’t believe I would have a good result with a 1000.000 features.
  • On the other hand, I set up a QGIS demo server and I could serve all my 200.000 features smoothly.

I guess that the difference is that using (Geo)JSON, there is too much work on the client side. Using WMS, the work load is on the server side and the Leaflet client only receives images.

It’s still going to give me some work to find out if I will be using QGIS Server, Geoserver or another WMS server and how I am going to implement the other wanted specifications.

Ok. Some update :

  • Using a WMS server seems to work very well in terms of integration with Leaflet. It was a bit more difficult to enable the functionality to have a popup opening when clicking on a GIS feature, but I worked out a first version of it now using “Better WMS” for Leaflet, which enables GetFeatureInfo requests.

I am now facing CORS issues (Cross-Origin Resource Sharing) because my Django http server is on http(s)://127.0.0.1:8000/ (for the moment, while in development) while the WMS server is at http(s)://qgis.demo/cgi-bin/qgis_mapserv.fcgi? ).
I read that a possible solution to CORS issues was to create a proxy with your http / application server so that resources from other domain names transit through your server, which serves everything to the client.

This sounds a great solution for me because if I can channel all my WMS queries through Django, I can at the same time control the users’ authentication and rights and log the activity using Django, which I have to do for other parts of the web site anyway.

The question is now : how do I implement such a proxy in Django ?

Eventually, implementing the proxy was quite straightforward :

def wmsproxy(request):
    assert isinstance(request, DjRequest)
    assert request.method == 'GET'

    params = [f'{k}={v}' for k, v in request.GET.items()]
    requesttype = request.GET.get('request')  

    httpresponse = httprequests.get(f'{WMS_SERVER_BASE_URL}?MAP={WMS_BASE_MAP}&' + '&'.join(params))

    django_response = DjResponse(
        content=httpresponse.content,
        status=httpresponse.status_code,
        content_type=httpresponse.headers['Content-Type']
    )
    return django_response