# Django Sitemap Best Practice

**URL:** <https://forum.djangoproject.com/t/django-sitemap-best-practice/18050>\
**Category:** Deployment\
**Created:** [January 7, 2023, 7:16am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050 "2023-01-07T07:16:16Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 7, 2023, 7:16am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/1 "2023-01-07T07:16:16Z")

</div>

# Django Sitemap Best Practice

I have about 200k urls. I set the per page limit to 5000, and page\_cache at 86400 (24 hours). The sitemap is dynamically generated by sitemap index ([The sitemap framework | Django documentation | Django](https://docs.djangoproject.com/en/4.1/ref/contrib/sitemaps/#creating-a-sitemap-index)). It is super helpful.

1 problem, `example.com/sitemap.xml` sometimes crash the server. The AWS EC2 console shows service are ok, but unable to ssh to the server, and web app is down.

I am still testing and not 100% sure about root cause, maybe

1. Internal factor, due to EC2, nginx, gunicorn or django
2. External factor, because of web crawlers (Googlebot)?

How do you ensure sitemap has high performance? What are the best practices if you have millions of urls? Or the only solution is to increase the server specs?

Would like to hear your experience.

## Sitemap items

```python
from django.contrib.sitemaps import Sitemap
from blog.models import Entry

class BlogSitemap(Sitemap):
    changefreq = "never"
    priority = 0.5

    def items(self):
        return Entry.objects.filter(is_draft=False)

    def lastmod(self, obj):
        return obj.pub_date

```

Is there anything we can do on the queryset to improve performance, such as selecting specific fields (urls, last\_modified). Or since it’s lazy, it’s already optimized?

## Cache

I am using page\_cache, are there any tips that would make a difference?

## Compress sitemap

1 technique is to create sitemap with `.gz` extension.

If we create the `.gz` files ourselves, how do we integrate that with the sitemap index?

## Cached vs compressed sitemap performance

Is compressed sitemap more performant than cached sitemap?

What are the pros and cons?

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 7, 2023, 12:52pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/2 "2023-01-07T12:52:28Z")

</div>

> [@shawnngtq](#):
>
> 1 problem, `example.com/sitemap.xml` sometimes crash the server. The AWS EC2 console shows service are ok, but unable to ssh to the server, and web app is down.

That indicates something _seriously_ wrong with your configuration.

In the absence of more detailed information, my best guess for this is that this is being caused by using too much memory.

What’s the actual size of the sitemap file(s)?

Does it come back eventually, or do you need to reboot the instance?

What size instance is this? Do you have a swap file allocated?

Are you running the database in that same instance?

Are you using something like cgroups or docker to set an upper limit on memory allocation by your Python processes? (I know that uwsgi has some settings to help facilitate that, I don’t know about gunicorn.)

Have you run this in a test environment such that you can determine a memory usage profile from it?

Is it possible for you to divide this large number of urls into sections?

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 7, 2023, 3:49pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/3 "2023-01-07T15:49:44Z")

</div>

I will get back with more details.

## Setup

1 server that host both django webapp and postgresql database.

I use filesystem caching. Yes, redis / memcached has better performance, as they use memory instead of FS. Will move to that eventually. Using FS now since storage is cheaper than memory.

## Server specs

When I was using AWS EC2 t3a.micro (1GB RAM), and execute `ping_google`, the server will hang and go offline.

Once I change it to t3a.small (2GB RAM), I can `ping_google` with no problem. Overnight it crash. The EC2 failed 1/2 instance checks. This is not conclusive, still need a few more days to observe.

If I delete the django cache and regenerate it manually, the server don’t crash.

Not using docker, direct deploy with gunicorn + uvicorn

## Sitemap

It is broken into 2 parts, static (\<10) and section1 (200k+, 5k per page)

```auto
sitemap.xml
  sitemap-static.xml
  sitemap-section1.xml?p=1
  ...
  sitemap-section1.xml?p=47

```

## Thoughts

I will load the data locally to do memory profile.

Haven’t done it yet since my local pc has 128GB memory, unlikely to see performance issue, and it’s tricky to test for gunicorn / uvicorn / nginx locally. Need to think how to do it.

I think t3a.small (2GB RAM) maybe the issue. The memory is not enough to share between postgresql, django and linux services. But my monitoring shows that it is not a problem.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/c/c0cebd18ad9e375019ccbc7b5442adde8c30f985.png)

Eventually, I will increase the sitemap urls to roughly 500M. Even if I set 50k per page, that will be 10k pages. Which is why I need to think about sitemap optimization, on top of increasing server memory.

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 7, 2023, 3:53pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/4 "2023-01-07T15:53:32Z")

</div>

> [@shawnngtq](#):
>
> Haven’t done it yet since my local pc has 128GB memory, unlikely to see performance issue, and it’s tricky to test for gunicorn / uvicorn / nginx locally.

Actually, that makes it easier. Just track the memory utilization of your processes. You’ve got enough “headroom” that you don’t need to worry about an out-of-memory situation.

And I’m not sure why you think it’s tricky to test locally - what operating system are you using for your development environment?

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 7, 2023, 3:56pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/5 "2023-01-07T15:56:49Z")

</div>

Using linux, I can use all linux distro, so env isn’t a issue for me.

Tricky since my PROD is the only real one. My local setup till date is to test only code and feature, I don’t load it with same data. I understand this is bad practice, I will start doing it properly.

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 7, 2023, 3:59pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/6 "2023-01-07T15:59:44Z")

</div>

Or, if you want to more replicate the production environment, you could use VirtualBox to create a constrained environment that more closely resembles the EC2 instance. That also provides the isolation you may want between this productionish system and your normal development environment.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 7, 2023, 4:05pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/7 "2023-01-07T16:05:29Z")

</div>

I will setup and do memory profile.

What really confuse me is the stats. It doesn’t look like it’s struggling.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/1/15b5d9618dc76655db1531427b17dc1f2bf427c9.png)

The maximum cpu usage is 50%. So CPU shouldn’t be bottleneck.

Memory usage maybe.

Is sitemap CPU or memory intensive for reading action?

What about generating FS cache, memory intensive?

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 2:45am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/8 "2023-01-08T02:45:57Z")

</div>

Most crude method

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/f/f37954d9c8933bf3096e1731b126540680b05572.jpeg)

My local pc is running on Unraid OS. I have multiple VMs and dockers.

This VM1 only host django webapp. Postgresql is hosted outside of VM1.

When I enter `VM1ip/sitemap.xml` on another VM2 browser

Memory usage jumps from 4.8GB to 7.4GB, about 2.6GB.

I guess for my AWS EC2 t3a.small (2GB), when I enter `sitemap.xml` for the 1st time, it probably will use 2.6GB (django webapp) + Postgresql memory usage (not recorded).

This is a challenge? If 200k urls need 3-4 GB RAM. Then 500M urls need 7500 - 10000 GB RAM?

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 8, 2023, 2:59am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/9 "2023-01-08T02:59:24Z")

</div>

Quite possibly.

I can’t imagine any valid reason from creating a sitemap file that large. I think you might want to rethink the site design if you have 500M non-parameterized unique URLs.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 3:00am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/10 "2023-01-08T03:00:44Z")

</div>

Yes, I can’t offer that kind of server.

I have that many url as I am using django to build a search engine.

I guess I shouldn’t add all to sitemap, need to be selective.

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 8, 2023, 3:02am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/11 "2023-01-08T03:02:46Z")

</div>

No, that’s not the purpose of sitemap. You need to fundamentally rethink your applications architecture.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 3:09am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/12 "2023-01-08T03:09:54Z")

</div>

![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/5/52d8441ec3a854874b9925e622b384d311fef9e3.png)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/f/f3c541b3693f27aaf50ad37517597603967d0295.png)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/3/3909224a8766b5c0d81370f2e2236abd99ab6c25.png)

I think you are right, other “search engine” don’t put all their urls in their sitemap.

Crunchbase has 11.3M urls, that’s roughly 169.5 - 226 GB RAM. I wonder if they dedicate a server just for sitemap.

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 8, 2023, 3:14am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/13 "2023-01-08T03:14:59Z")

</div>

Why do you think that those sites actually create a “sitemap” file? What you’re showing doesn’t in any way imply that they create an xml file with this data.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 3:34am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/14 "2023-01-08T03:34:46Z")

</div>

Crunchbase: [https://www.crunchbase.com/www-sitemaps/sitemap-index.xml](https://www.crunchbase.com/www-sitemaps/sitemap-index.xml)

The sitemap pattern is very similar to django sitemap index, just that they use compressed file (.gz) instead of `?p=`

```auto
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">
<sitemap>
<loc>https://www.crunchbase.com/www-sitemaps/sitemap-acquisitions-0.xml.gz</loc>
<lastmod>2023-01-08T00:08:38.000Z</lastmod>
</sitemap>
<sitemap>
<loc>https://www.crunchbase.com/www-sitemaps/sitemap-acquisitions-1.xml.gz</loc>
<lastmod>2023-01-08T00:08:41.000Z</lastmod>
</sitemap>
<sitemap>
<loc>https://www.crunchbase.com/www-sitemaps/sitemap-acquisitions-2.xml.gz</loc>
<lastmod>2023-01-08T00:08:44.000Z</lastmod>
</sitemap>
...
</sitemapindex>

```

I think crunchbase use cron job to generate gz file, and maybe they don’t update older gz files.

If I follow this pattern, I shouldn’t use django sitemap index. I should create the gz files myself, then somehow let sitemap reference them.

I will close this since this is more generic and not django specific.

---

<div class="post-metadata">

**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\
**Post date:** [January 8, 2023, 3:41am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/15 "2023-01-08T03:41:08Z")

</div>

The fact that [crunchbase.com](http://crunchbase.com) _sends_ you a file from a url named `sitemap-index.xml` does _not_ mean that they generate and store that as an xml file. It could just as easily be generated on the fly using the equivalent of a Django view from data stored in their database.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 3:49am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/16 "2023-01-08T03:49:30Z")

</div>

Good point, I am doing more research, maybe can be done using django view like you say.

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [January 8, 2023, 10:27am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/17 "2023-01-08T10:27:23Z")

</div>

![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/4/40157dfb4949391044aca179186a785376576f6d.png)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/djangoproject/original/2X/d/dd51f210c5805ca1dedacd2b1f785afa4852276d.jpeg)

EC2 hanged again 😴, CPU usage will suddenly spike before that.

Quite sure must be caused by bots and sitemap.

I will find a proper solution soon

---

<div class="post-metadata">

**Author:** ![anefta](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/anefta/32/10649_2.png) [@anefta](https://forum.djangoproject.com/u/anefta)\
**Post date:** [September 4, 2023, 10:58am UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/18 "2023-09-04T10:58:59Z")

</div>

Hi, did you find a solution?

---

<div class="post-metadata">

**Author:** ![shawnngtq](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/shawnngtq/32/5834_2.png) [@shawnngtq](https://forum.djangoproject.com/u/shawnngtq)\
**Post date:** [September 17, 2023, 12:46pm UTC](https://forum.djangoproject.com/t/django-sitemap-best-practice/18050/19 "2023-09-17T12:46:10Z")

</div>

@anefta , the temporarily solution is to increase memory.

The EC2 logs indicate that is memory issue. I changed the EC2 instance from micro to small. It will still hang once it a while, but manageable.
