Stop the Django 5.0 press: Move `URLField.assume_scheme` change from deprecation to hard-cut

On Django 4.2 and below, formsURLField adds the “http” scheme to scheme-less URLs. For example, it cleans example.com into http://example.com. Ticket 34380 makes this assumption https://example.com.

In the related PR, the implementation initially went for a hard cut, changing the field to assume “https” immediately. But this was changed to a deprecation after this comment by David Wobrock, stating it was backwards-incompatible.

I propose moving the implementation back to a hard cut approach before Django 5.0 has its impending final release.

I base this proposal on my experience fixing the warning in a project that I was testing on Django 5.0. I started drafting a blog post covering the multiple required techniques to set assume_scheme everywhere, covered at the end of this post. Whilst writing, I realized it’s a lot of work for such a minor change, and I think we can avoid hoisting this work on the community.

I would guess that most users entering a URL into a form copy-and-paste it from their browser, in which case it comes with a scheme. For the remaining cases where a URL comes with no scheme, it’s still overwhelmingly likely to work with HTTPS. According to Google Chrome’s statistics (“HTTPS Encryption by Chrome platform”), 93% of page loads on Windows use HTTPS.

Using the deprecation pathway to make this change adds warnings to projects for every URLField, including those generated by ModelForm, ModelAdmin, and third-party packages. It helps only a small number of projects that rely on users entering HTTP URLs without schemes (corporate intranets are my prime suspect).

I think we can make a hard cut without much problem. As a precedent, the now-removed ping_google command was hard cut to use HTTPS in Ticket 23829.

Sorry for the many words. Tagging people involved in the ticket: @felixxm.


Techniques from my drafted blog post:

Plain Forms with a URLField wrapper

For forms that you directly create, add the assume_scheme argument to their URLFields:

from django import forms


class CheckForm(forms.Form):
    url = forms.URLField(label="URL to check", assume_scheme="https")

If you have many URLFields, try functools.partial to reduce the repetition:

from functools import partial

from django import forms


HTTPSURLField = partial(forms.URLField, assume_scheme="https")


class CheckForm(forms.Form):
    url = HTTPSURLField(label="URL to check")

ModelForms with formfield_callback

ModelForms are a bit more tricky since their fields are automatically created “behind the scenes” from model fields. You can override these with explicit form fields, but that loses the automatic synchronization of other arguments like max_length.

Instead, you can use formfield_callback, added in Django 4.2. This callback hook allows you to wrap the creation of form fields to make adjustments as necessary. Use it to add assume_scheme=“https” to URLFields like so:

from django import forms
from django.db import models

from example.models import NicheMuseum


def urlfields_assume_https(db_field, **kwargs):
    """
    ModelForm.Meta.formfield_callback function to assume HTTPS for scheme-less
    domains in URLFields.
    """
    if isinstance(db_field, models.URLField):
        kwargs["assume_scheme"] = "https"
    return db_field.formfield(**kwargs)


class NicheMuseumForm(forms.ModelForm):
    class Meta:
        model = NicheMuseum
        fields = ["name", "website"]
        formfield_callback = urlfields_assume_https

For forms that already use formfield_callback, you’ll need to merge functions as appropriate.

ModelAdmin form field method override

If you’re using Django’s admin site, that’s another source of outdated URLFields. ModelAdmin classes automatically generate ModelForm classes, unless a form is explicitly declared. Because this form class is created per-request, the warning won’t appear at import time, but when you load the related “add” or “change” pages:

$ python -Xdev manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

...
/.../django/db/models/fields/__init__.py:1140: RemovedInDjango60Warning: The default scheme will be changed from 'http' to 'https' in Django 6.0. Pass the forms.URLField.assume_scheme argument to silence this warning.
  return form_class(**defaults)
[25/Nov/2023 16:37:34] "GET /admin/example/nichemuseum/add/ HTTP/1.1" 200 8907
...

You can also expose the warnings in your test suite by testing all of your ModelAdmin classes. See my previous post on using parameterized tests to do this easily.

The easiest way to fix these warnings is with the undocumented ModelAdmin.formfield_for_dbfield() method. This is passed as ModelForm.formfield_callback, and applies some customizations for the admin site. Override it in custom admin classes that you then use as bases in your project:

from django.contrib import admin
from django.contrib.admin.options import BaseModelAdmin
from django.db import models

from example.models import NicheMuseum

# Custom admin classes for project-level modifications


class AdminMixin(BaseModelAdmin):
    def formfield_for_dbfield(self, db_field, request, **kwargs):
        """
        Assume HTTPS for scheme-less domains pasted into URLFields.
        """
        if isinstance(db_field, models.URLField):
            kwargs["assume_scheme"] = "https"
        return super().formfield_for_dbfield(db_field, request, **kwargs)


class ModelAdmin(AdminMixin, admin.ModelAdmin):
    pass


class StackedInline(AdminMixin, admin.StackedInline):
    pass


class TabularInline(AdminMixin, admin.TabularInline):
    pass


# Admin classes


@admin.register(NicheMuseum)
class NicheMuseumAdmin(ModelAdmin):
    pass

Third-party packages

URLFields in third-party packages may be hard to fix in your project. Some packages do allow you to replace their form classes, so you can use the above techniques. Others don’t, so you may need to resort to monkey-patching.

In either case, projects will hopefully accept pull requests to use assume_scheme=“https”, if appropriate. Packages can pass the argument on Django 5.0+ by using this pattern:

from functools import partial

import django
from django import forms

if django.VERSION >= (5, 0):
    URLField = partial(forms.URLField, assume_scheme="https")
else:
    URLField = forms.URLField


class CheckForm(forms.Form):
    url = URLField(label="URL to check")
2 Likes

Hey Adam. Thanks for raising this.

Given what you’ve said, I think a clean break is likely the lesser of two evils. So yes.

The option that comes quickly to mind might be a transitional setting, so you can toggle it in one place, rather than every field instance. :thinking:

TBH I’m not seeing the value of that initially. Are plain http URLs still so common we need to jump through lots of hoops to support not typing the scheme for them in late 2023? (Maybe they are… IDK… :woman_shrugging: I might be more disposed to it after the second :coffee: but it doesn’t seem like explicitly specify the http scheme if you need it is too much of a burden.)

2nd :coffee: Update: Maybe that setting… :sweat_smile:

Thanks for creating this thread, however, I don’t see how this significantly differs from similar deprecations we’ve done in the past, e.g.

and why we should break our policy here :thinking:.

The point I find compelling is that there’s no once and you’re done migration path. You basically have to adjust every field, every form, every admin. That’s a lot of inconvenience. (Folks will not be happy: it’s not easy upgrade.)

Hence my thought about a setting: toggle it once and you’re done. :thinking:

I think the biggest difference is this applies to user-facing data entry rather than the “project-level” cases you cited. Changing assumptions around the current project’s scheme is different to assumptions around arbitrary URLs.

I think previous changes to form fields are closer analogies. I looked back and found these, none of which went through the deprecation pathway:

Furthermore, we’re still leaving a way to provide data in the same format: enter URLs with the “http” scheme. And projects can preserve the old behaviour by setting assume_scheme="http" where necessary.

A “no once and done” path could be a compromise. Maybe setting a class-level URLField.assume_scheme variable rather than a setting? But I am not convinced it’s even worth the consideration of most users.

Thank you @adamchainz and everyone else for the detailed proposition and feedback. This is a tough one!

I absolutely see the pain in (potentially) having to update lots of lines of code with this change. A naïve search in Github for forms.URLField returned 8.8K matches! :scream:
What’s worse, and as you said, majority of users having to make this change would very likely be OK with https being the default, so we are (sort of) forcing extra work on this group, like a (form of) punishment for a code that “did nothing incorrectly”.

At the same time, I think we are too late in the Django 5.0 schedule to change this from warning to hard cut. If this would be the other way around (going from hard cut to deprecation), or if this would be occurring before the release candidate deadline, I would be totally on board. But now, a week before the final release, it just feels wrong and too risky to change semantics like proposed.

I’m +1 for providing a simpler/“once and done” way of acknowledging the new scheme default, and after some thinking and brainstorming, I think a new setting like proposed by @carltongibson is the safer and simpler way to do it. Concretely, a new setting (name to be agreed) DEFAULT_URLFIELD_SCHEME, defaulting to None. The setting would also be transitional and removed in 6.0. Then, in the URLField, something like this would work I think:

diff --git a/django/forms/fields.py b/django/forms/fields.py
index d1ba8af654..d3da9d2a5f 100644
--- a/django/forms/fields.py
+++ b/django/forms/fields.py
@@ -15,6 +15,7 @@ from decimal import Decimal, DecimalException
 from io import BytesIO
 from urllib.parse import urlsplit, urlunsplit
 
+from django.conf import settings
 from django.core import validators
 from django.core.exceptions import ValidationError
 from django.forms.boundfield import BoundField
@@ -761,11 +762,13 @@ class URLField(CharField):
     default_validators = [validators.URLValidator()]
 
     def __init__(self, *, assume_scheme=None, **kwargs):
+        assume_scheme = assume_scheme or getattr(settings, "DEFAULT_URLFIELD_SCHEME", None)
         if assume_scheme is None:
             warnings.warn(
                 "The default scheme will be changed from 'http' to 'https' in Django "
                 "6.0. Pass the forms.URLField.assume_scheme argument to silence this "
-                "warning.",
+                "warning, or define the setting DEFAULT_URLFIELD_SCHEME to be the "
+                "default scheme when validating this field.",
                 RemovedInDjango60Warning,
                 stacklevel=2,
             )

The class-level attribute proposed by @adamchainz could also work but it forces users that “did nothing wrong” to define a new class and use that across their code base.

If we are doing this change (I see the value in doing it), I’d like us to make a decision and propose a PR ASAP given the proximity of the final release.

Thank you, Natalia.

1 Like

Disclaimer I’ve only scanned over the arguments but I +1 with Adam.

Another option I could think of would be completely reverting the revision and re-assessing the plan forward for 5.1?

Link to further conversations in the Discord contributor chat about this.

That vastly underestimates the number of URLFields in forms because most will come via model forms.

I get that we want to be conservative. I do think that very few people have built any kind of dependency on the change by release candidate stage though. This mostly affects projects rather than third-party packages, whilst I think the majority of testing before release is of third-party packages.

I meant more like:

from django import forms
forms.URLField.assume_scheme = "https"

Asking users to subclass would have the same problems as the existing solution: all forms would need updating.

That said maybe a setting would be better as it’s the general migration mechanism in Django, and we could enable it in the new project template.

—-

My ranked preferences (edited):

  1. Change default to https
  2. Setting (defaulting to https in the startproject template)
  3. Class-level attribute
  4. Revert and reconsider
1 Like

I don’t think it can be the case that we can’t make changes between rc1 and final — the whole point of the pre-release phase is to reveal these kind of issues.

I don’t like a transitional setting very much, but we’ve done similar a number of times in order to avoid a breaking change. I think we have to stick to the API stability policy, even in cases where we might be tempted to skip it.

I’d lean that way rather than revert, since this is a good change — we should keep it.

This could be a transitional setting, however we will not be able to remove it in Django 6.0, because a setting deprecated in Django 5.0 and removed in Django 6.0 will create exactly the same issue as the current deprecation. Folks will get a deprecation warning when using it.

Is it not enough to ignore the deprecation warning if you don’t mind a new default? I’m not sure why we assumed that folks are forced to update anything :thinking:

Yes, but we introduce them for a short period of time when folks were updating multi-instance apps, not to silence a deprecation warning and adopt the new default.

Then at the least we should document the exact warning filter one would need to add – but doing this still leaves no simple path to Opt-in to the future — we’re not giving folks an easy way to do the right thing.

This, I do not understand :thinking:

I was proposing that in 6.0 we remove the setting and make the assume_scheme default to https. Users needing to use another scheme for URLField other than https should explicitly use the assume_scheme param and not the setting for this. Having said this, perhaps the transitional setting would be better named as DEFAULT_URLFIELD_SCHEME_HTTPS

The only difference is that with a transitional setting they will adopt to the new default (“https”), but this setting would also be deprecated so they would be forced to ignore deprecation warning (if they care) or don’t use the setting and we are again in the same starting point.

As far as I’m aware, ignoring a deprecation warning is not an issue. The main issue is how to smoothly adopt to the new default before the deprecation ends.

Yes, we need to give folks an easy way to Opt in to the future.

I think change the default, add a transitional setting to maintain the old behaviour (if needed, for now) and deprecate that immediately. (If you want to keep http for now, you get the warning that it’s going away.)

This is exactly what we did with USE_DEPRECATED_PYTZ no?

Let me propose something tomorrow :bulb: :brain:

2 Likes
1 Like