Custom encrypted JSONField breaks when moving from 4.1 to 4.2

Using Django Mystery Errors
1

Hey there, I have my own custom encrypted JSON that looks like this:

class EncryptedJSONField(models.JSONField):
    def to_python(self, value):
        return value

    def get_db_prep_save(self, value, connection):
        value = super().get_db_prep_value(value, connection, prepared=False)

        if value is None:
            return value

        parsed_value = json.loads(value)

        for key in parsed_value:
            parsed_value[key] = signing.dumps(parsed_value[key])

        return json.dumps(parsed_value)

    def from_db_value(self, value, *args, **kwargs):
        if value is None:
            return value
        parsed_value = json.loads(value)
        for key in parsed_value:
            parsed_value[key] = decrypt_value(parsed_value[key])

        return parsed_value

After upgrading from Django 4.1 to 4.2 it doesn’t work the same, it looks like it encrypting my value twice. I thought my issue might be similar to this one: #34539 (`get_prep_value` no longer called for JSONField) – Django - but that was fixed in 4.2.2 and doesn’t fix my issue.

I first had to change 1 thing

value = super().get_db_prep_value(value, connection, prepared=False)

This now returns a <psycopg2._json.Json object, where it returned a string before. So I changed parsed_value = json.loads(value) to parsed_value = value.adapted to convert it to a python object.

So my get_db_prep_save looks like this now:

    def get_db_prep_save(self, value, connection):
        value = super().get_db_prep_value(value, connection, prepared=False)

        if value is None:
            return value

        parsed_value = value.adapted

        for key in parsed_value:
            parsed_value[key] = signing.dumps(parsed_value[key])

        return json.dumps(parsed_value)

But after I do that, it seems like the it’s encrypting twice. Here is an example model:

class Authentication(models.Model):
    details = EncryptedJSONField(blank=True, null=True)

And here is a simple test that works on 4.1.13 but fails on 4.2.11

    def test_encrypted_field(self):
        details = {"username": "foo", "password": "bar"}
        authentication = Authentication.objects.create(details=details)
        assert authentication.details["password"] == "bar"

The error is AssertionError: assert 'IkltSmhjaUk6...mtQzkCHT6NQK4' == 'bar' (Which is the value if I run it through signing.dumps twice) Does anyone have an idea on how to fix this?

2

Hi, can you share the implementation of the decrypt_value function ?

Side note: you use the term encrypting, but here you never encrypt values, they are just signed which is not the same thing. Anyone with access to the database can read the original values without knowing the secret key.

3

Hi, here is the decrypt implementation:

def decrypt_value(value: str) -> str:
    try:
        return signing.loads(value)
    except signing.BadSignature:
        return value

But you are right, it is just signing, not encrypting, but it is signing using django.core.signing (Cryptographic signing | Django documentation | Django), which does indeed use the SECRET_KEY. Am I wrong in thinking they would need the SECRET_KEY to know the original value if they had access to the database?

4

Yes, you’re wrong thinking that. If you take the first part (before the first “:” character) of the encoded value, and you base64 decode it, you’ll see the original value. Signing the value just ensures it was generated/validated by your project (i.e. the owner of the secret key).

Now, regarding the problem you have: when decrypting, if you get a BadSignature exception, you just ignore it and return the signed value. I don’t know if this is what’s wrong here, but if you are in such case, that may lead to double encode the original value if it’s
encoded once and saved to db, then reloaded but not decoded (because of bad signature), then saved again to database, which will issue a new encoding (leading to double encoding).

We have to make sure where the problem lies. I don’t know why you catch the BadSignature exception (if thevvalue cannot be verified, then there is a problem and returning the stored value will necessarilly lead to a bad value), but at least here, you may add some log/print statement in the try/except clauses so that we can know whether the problem is a signature verification problem or not.

1 Like