Handling German Umlaute when writing to JSONField to database

Using Django Forms & APIs
1

I am using an existing database which is encoded in utf8mb4_general_ci.
The problem I am facing occurs with postgres and SQLite:

my database:

class Product(models.Model):
    data = models.JSONField()
    number = models.PositiveIntegerField()

when writing the word Gemüse to the db, I can see this entry in SQLite:

{"name": "Gem\u00fcse"}

This is a problem, because I have a search function on the site, that selects sets from the database like this:

def get_queryset(self):
        qs = super().get_queryset().order_by("number")
        search = self.request.GET.get("search")
        if search:
            qs = qs.filter(Q(data__icontains = search) | Q(number__icontains = search))
        return qs

and obviously searching for Gemüse yields nothing.
I am asking myself what my best way forward is here:

  • Should I try to save my data to the database the same way as now and modify my search so I can find Gemüse also as Gem\u00fcse? How would I do that?
  • Is there a way to transform my database from utf8mb4_general_ci to another encoding so I can get rid of the problem this way? This is only important for the postgres part, the SQLite ist just for testing
  • Could I already implement something on the template level to make sure this problem never comes back?
2

Maybe a little bit of an Add-on as this is not the first trouble I ran into:

Before the update from django 3 → 4 I used a customized JSONField:

class CustomJSONField(models.JSONField):
    def get_prep_value(self, value):
        if value is None:
            return value
        return json.dumps(value, ensure_ascii = False)

class Product(models.Model):
    data = CustomJSONField()

But after the update (and a server update as well) I got all my JSON data into the database like this: "{\"name\": \"Gem\u00fcse\"}". So I thought I can maybe change that back to just using JSONField but in testing the Umlaute-Error occured (but I got rid of the escaped quotes and got a valid JSON and not a string).