Authentication wall for staging sub domain to aid SEO

I am trying to follow best SEO practices and one of the things I am trying to avoid is duplicate content issues with a staging environment.

My production site is aa-london.com and my staging site is staging.aa-london.com

From what I have read it is best practice to use authentication and wall off the whole staging site.

I came across django-auth-wall which looks a bit old.

Is there an easy way in my settings file that I can configure django to require auth for all views?

Thanks
Chris

I can’t think of any setting that would quickly enable all views to require authentication. If I think about it, how would a setting like that permit someone to login if the login view also required auth? :grin:

For your use case, since you’re concerned about SEO, I think a good solution would be to have a proper robots.txt file. If you include a robots.txt with the following content on your staging site, then the crawlers for the big websites that really impact SEO (namely, Google) will respect that and not crawl the staging site.

User-agent: *
Disallow: /

If I were in you, I would make a custom view that would serve robots.txt with a setting for your environment. Assuming you add the setting to the context (which I’m naming ALLOW_CRAWLING in this example), the template could look something like:

User-agent: *
{% if ALLOW_CRAWLING %}
Allow: /
{% else %}
Disallow: /
{% endif %}

I hope that helps!

2 Likes

Thanks!

I was reading this link https://searchengineland.com/how-to-keep-staging-development-site-index-286987
which was saying use authentication as a prefered option over robots.txt, so just wondered if anyone did this kind of thing before or if robots.txt change is correct.