ModelForm with field depending on user permission?

I have a model Address with a field and a ModelForm.

class Address(Model):
    number = PositiveSmallIntegerField(default=1)
class AddressForm(ModelForm):
    class Meta:
        model = Address
        fields = "__all__"

How do I add a validator to it, when the form is shown to a user with a specific permission? The form should behave as if the model itself would have a number field like this:

number = PositiveSmallIntegerField(default=1, validators=[MaxValueValidator(100)])

By that I mean that the form should only allow values from 0 to 100, and the backend should validate that the value is inbetween 0 and 100.

The view could pass whether or not to use the limited field like so:

form = AddressForm(enable_limit=True)

And I could do something like this:

class AddressForm(ModelForm):
    class Meta:
        model = Address
        fields = "__all__"

    def __init__(self, *args, enable_limit=False, **kwargs):
        if enable_limit:
            self.base_fields["number"].widget.attrs["max"] = 100

However, this only updates the widget. The form does complain about values > 100, but I can go “hack” my way around that and would still not have a validation on the backend.

How would you solve this?