Testing Django's new JSONField

Django Internals Mentorship
1

Hi All,

I’m using django.contrib.postgres.fields JSONField and have a test suite that and some time to test the new JSONField from https://github.com/django/django/pull/12392

Now, at the risk of sounding daft, I have to ask, how do I get this new code on my laptop? There’s a PR in Github, that’s all well and very good, but how do I get that into my Django 3.0 that I’ve installed from PyPi with Pip? My guess is that I need to clone Django, switch to the branch in question, fire it up and test my Django project. Even if I’m correct, which I fear I’m not, I’m still a bit confused about the process as a whole.

So my question is really this: is there a link or document that someone can point me to that explains this process of how we go about testing a PR? Or perhaps there is someone who can put me back on the straight and narrow?

Apologies if this is all bit straightforward and I’ve missed something obvious. I saw something on twitter, and thought to myself, ‘yep, I can do that’, and then quickly found myself googling, ‘hmm, how do I do this?’

Cheers,

C

1 Like
2

Hi @conor. Good question.

So…

Yes, clone django. I suggest using the --depth option, unless you really want the whole history…

$ git clone --depth=50 https://github.com/django/django

Then you´ll want to checkout the PR branch. You could do that by adding Sage´s fork as a remote, and checking out that but there´s a nicer way:

$ git fetch origin pull/12392/head:jsonfield && git checkout jsonfield

That fetches the PR into a new branch and checks it out in one.

Then create a new venv and install your requirements. Then install your Django clone into your venv.
With the venv active:

$ cd <your/django/clone>
$ pip install -e .

That will install Django, checked out at the PR branch in your venv and you can start editing your project and running your test suite. (Perhaps do that first just to check.)

Thanks for taking the time! Let us know if you hit issues.

Carlton

1 Like
3

Cheers @carltongibson,

Very good of you to respond, thank you!

I fumbled my way through cloning the branch directly from laymonage’s repo with

$ git clone git@github.com:laymonage/django.git --branch ticket_12990

But your method will definitely save me some time in the future and give me access to all recent PR’s. Always nice to learn new tricks. Cheers for that!

The short version is that I’ve got it up and running and everything looks good and my tests run well. If all my tests run well and I find no issues, is that something worth posting on Github, or only wasting the devs’ time? Only too happy to comment on Github if appropriate.

For the long version, keep reading (it has a happy ending).

After cloning the repo I went about the venv dance, cloned my app and fired it up after modifying my models. (Pro tip: remove Django from your own repo’s requirements.txt before installing your requirements! That had me scratching my head for a while)

Regarding the JSONField, I can report as follows:

Changed

from django.db import models
from django.contrib.postgres.fields import JSONField
...
...
class ModelOne(models.Model):
    my_attribute = JSONField()

class ModelTwo(models.Model):
    my_attribute = JSONField()

to (deliberately only changed one of the attributes to the new JSONField)

from django.db import models
from django.contrib.postgres.fields import JSONField
...
...
class ModelOne(models.Model):
    my_attribute = models.JSONField()

class ModelTwo(models.Model):
    my_attribute = JSONField()

$ python manage.py makemigrations
System check identified some issues:

WARNINGS:
progress_tracking.TestsRunTracking.results: (fields.W903) django.contrib.postgres.fields.JSONField is deprecated and will be removed in Django 4.0.
	HINT: Use django.db.models.JSONField instead.
No changes detected

I then removed all references to django.contrib.postgres.fields import JSONField from my models.py:

from django.db import models
...
...
class ModelOne(models.Model):
    my_attribute = models.JSONField()

class ModelTwo(models.Model):
    my_attribute = models.JSONField()

$ python manage.py makemigrations
No changes detected

I then ran my tests and got a lovely long line of ... and

Ran 111 tests in 4.135s

OK

I’ll do some manual testing today and tomorrow and let you know how it goes.

Again, thanks for the help!

Cheers,

Conor

1 Like
4

Super Conor. Thanks. I think posting “All good”, perhaps linking to more details here is worth it. Otherwise we just think no one ran it. :grinning:

Please do do the manual testing too. Thanks. :+1:

1 Like
5

That is good news. Thanks for testing and sharing your experience, Conor! :smile:

Oh, and while you’re at it: perhaps try using SQLite and see if everything works as well? SQLite should be pretty easy to setup (it’s the default after all).

6

You’re very welcome!

It’ll be straightforward enough for me to test SQLite as I have an importer to populate the DB and some scripts which interact with the site which in turn is what generates records containing JSONFields. I’ll give it a go this evening after my son is asleep. Friday nights just aren’t the same anymore!

On another note, my app doesn’t use the JSONFields for any sort of querying, they’re stored only really as a record of dynamic results that were previously returned. My question for you is, on a scale of 1 to 10, how useful would it be for you if I were to write some basic tests which ran queries against the DB using the JSONField as part of a queryset’s filter?

1 Like
7

I think that depends on the queries and data. The simple cases are already covered in the tests in my PR. However, I think running some tests on an actual project would still be beneficial. On a scale of 1 to 10, I can’t say for sure… I’d say it’s always useful to know what the test results are (across different setups), so maybe it’s 8, at least? However, you shouldn’t push yourself too hard if it isn’t necessary for your project. I think it’s more useful to test the lookups if you’ve got a use case where the queries are actually needed because it’s more natural.

8

G’day Sage,

I can report that my test suite and the script below were performed with both Postgres 11 and SQLite 3.28.0. Postgres was running in Docker, SQLite on my local MacBook Pro OSX 10.15.2. Your branch of Django was run with Python 3.8.1 locally on my Macbook.

I have a model which looks like this:

class CaseSession(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    user = models.ForeignKey(
        CustomUser, related_name="user_sessions", on_delete=models.CASCADE
    )
    case_attempt = models.ForeignKey(CaseAttempt, on_delete=models.CASCADE)
    session = models.JSONField()

I have an endpoint with a DRF read-write model serialiser. My test suite tests POST, GET and PUT against this endpoint. The entire test suite runs without a problem.

Furthermore, I have a small script which interacts with the API in much the same way a user might interact with it via the JS frontend. Here, much like my test suite, I create a CaseSession object with a POST, retrieve it with a GET and modify it with a PUT.

The POST request creates an object with:

# POST request
session = create_sesssion(user_id, case_attempt)
print(json.dumps(session["session"], indent=4, sort_keys=True))

{
    "amp": 11,
    "hey": "joe",
    "key": "value"
}

The PUT requests updates this object to:

# PUT request to the id of the session returned by the above POST call
session = update_session(user_id, session["id"], case_attempt)
print(json.dumps(session["session"], indent=4, sort_keys=True))

{
    "hey": "updated joe",
    "key": "updated value",
    "nested": {
        "blackbird": "tree branch",
        "firecrest": "first time breeders in Oslo, 2019",
        "pied flycatcher": "hopefully in the birdhouse I built",
        "woodpecker": "tree trunk"
    },
    "volume": 11
}

And the GET request returns the exact same as what was returned by the PUT request

# GET request to the id of the session returned by the above PUT call
session = get_session(session["id"])
print(json.dumps(session["session"], indent=4, sort_keys=True))

{
    "hey": "updated joe",
    "key": "updated value",
    "nested": {
        "blackbird": "tree branch",
        "firecrest": "first time breeders in Oslo, 2019",
        "pied flycatcher": "hopefully in the birdhouse I built",
        "woodpecker": "tree trunk"
    },
    "volume": 11
}

Now that the HTTP requests all look good, lets run a couple of calls to my CaseSession model using filter() on the queryset

# get a CaseSession by filtering on the JSONField key 'volume'
test_data = CaseSession.objects.all().filter(session__volume=11).first()
print(f"Amplifier max volume: {test_data.session['volume']}")

Amplifier max volume: 11

And finally run a query going into the nested JSON and using __contains:

# get a CaseSession by filtering on the JSONField key 'nested.firecrest'
test_data = CaseSession.objects.all().filter(
		session__nested__firecrest__contains="Oslo"
).first()
print(f"Firecrest nesting: {test_data.session['nested']['firecrest']}")

Firecrest nesting: first time breeders in Oslo, 2019

I also have another model which uses a JSONField for storing, believe it or not, medical results from an imaginary bunny rabbit who is being treated in a vet’s clinic (this is a learning app for vet students).

As part of the script that was used above, I interact with endpoints that return results that are in turn stored in another model in JSON. These results are retrievable with a GET request and are returned as expected.

Righty oh, I hope this helps, and if you have some examples of other bits and pieces you would like me to test, please don’t hesitate to say so, and I’ll see what I can do.

And last, but by no means least, thank you @sage for developing this; good on you and well done! Three big cheers!

1 Like
9

Thank you very much for sharing your experience (and very thoroughly), Conor! I guess we need people to test the field on MySQL, MariaDB, and Oracle, but it’s a bit of a hassle if you don’t normally use it. If you want to continue testing on PostgreSQL and SQLite, there are some more complex queries that can be done by using lookups and transforms, as demonstrated in the tests. I’m quite confident they work though, so it’s okay if you want to skip them :smiley:

10

You’re welcome, Sage. I tried getting my app up and running with MySQL yesterday evening. Managed to apply migrations without an issue, but when running my tests, the whole stack fell apart. I haven’t localised the issue yet, but will give it another shot this evening.

11

Hello again. OK, so I got my stack running for both MySQL and MariaDB. Long story short, both Django and database are in docker in their own separate containers. Again, Python 3.8.

MariaDB: 10.4.12
MySQL: 8.0.17

I performed the same tests as I did in my previous post.

Results

Passing

  • Migrations: OK on both
  • My own test script (not Django test suite) OK on Both
  • Running my own database importer against MySQL, not MariaDB. It should be pointed out that I believe this has nothing to do with JSONField, as my importer tool doesn’t import any data that is stored as JSON, nor does it create any records in a Model which have a JSONField attribute

Failing

My Django test suite fails on both MariaDB and MySQL. Again, I think this is a problem with my stack, and not JSONField. It appears I need to install GDAL into my Django docker image, but that will have to wait for another day. It should be pointed out that my test cases are not failing, but rather a

======================================================================
ERROR: django.contrib.gis.utils (unittest.loader._FailedTest)
----------------------------------------------------------------------

Full stack trace below.

Interestingly enough, my data importer didn’t work with MariaDB. It failed about half way through and threw the following exception (again, this appears to have nothing to do with JSONField)

django.db.utils.OperationalError: (1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation '='")

Summary

Given that both databases successfully handled my own custom testing script (again, not Django tests), I’m quite confident that JSONField worked as expected and as tested by my script.

I suspect that the failures are due to missing requirements and possibly a locale setting in my test stack. I’ll see if I can iron them out sometime in the near future.

Last, I will try and write some more advanced queries as suggested by Sage should I get the chance in the near future. Who knows, I might learning something!

Stack Traces

Django test cases failing

======================================================================
ERROR: django.contrib.gis.utils (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: django.contrib.gis.utils
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/local/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/code/django/contrib/gis/utils/__init__.py", line 4, in <module>
    from django.contrib.gis.utils.ogrinfo import ogrinfo  # NOQA
  File "/code/django/contrib/gis/utils/ogrinfo.py", line 7, in <module>
    from django.contrib.gis.gdal import DataSource
  File "/code/django/contrib/gis/gdal/__init__.py", line 28, in <module>
    from django.contrib.gis.gdal.datasource import DataSource
  File "/code/django/contrib/gis/gdal/datasource.py", line 39, in <module>
    from django.contrib.gis.gdal.driver import Driver
  File "/code/django/contrib/gis/gdal/driver.py", line 5, in <module>
    from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
  File "/code/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
    from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
  File "/code/django/contrib/gis/gdal/libgdal.py", line 39, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Could not find the GDAL library (tried "gdal", "GDAL", "gdal2.4.0", "gdal2.3.0", "gdal2.2.0", "gdal2.1.0", "gdal2.0.0"). Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.

Locale issue whilst importing data into MariaDB

Traceback (most recent call last):
  File "/code/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/code/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation '='")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "importer.py", line 73, in <module>
    main()
  File "importer.py", line 63, in main
    import_diagnostics(diagnostics, parameters, CASES)
  File "/code/tools/importer/diagnostics.py", line 37, in import_diagnostics
    base_values = create_base_blood_result(data)
  File "/code/tools/importer/diagnostics.py", line 291, in create_base_blood_result
    not BaseResult.objects.all()
  File "/code/django/db/models/query.py", line 794, in exists
    return self.query.has_results(using=self.db)
  File "/code/django/db/models/sql/query.py", line 536, in has_results
    return compiler.has_results()
  File "/code/django/db/models/sql/compiler.py", line 1103, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/code/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/code/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/code/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/code/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/code/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/code/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/code/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/code/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation '='")
1 Like
12

Just one thing to note (sorry for distracting from the general discussion), it might be simpler to suggest something like this in the future:

pip install --no-cache --upgrade https://github.com/django/django/archive/pull/12392/head.zip

To me this is significantly less friction, which is important for getting people to try something. Updates can be pulled by just re-executing the installation command and it can be pretty easily added to an existing project’s requirements files for testing (just replace django==... with that URL).

2 Likes
13

Yes, pip’s VCS support is first rate. No idea why that didn’t occur to me as the best option the other day. (Habit no doubt…) :+1:

3 Likes
14

Hi @sage, @carltongibson,

I was just catching up on the Chat Django podcast and am currently listening to you two chat, and I thought, crikey, I’ve bloody well dropped the ball on this one. Very sorry about that, lads. Really, mea culpa.

As we probably all are, I’m very busy with work and being home with a three year old (don’t know how you do it, Carlton, but good on you!) so finding time for other things is a bit tricky right now. However, if you think there is something more with which I can help, don’t be afraid to write it here and I’ll give you some honest feedback of how I may or may not be able to help.

Again, very sorry for completely forgetting about this one.

All the best, and please, stay healthy!

Conor

2 Likes
15

Hey Conor. Thanks for your words. Don’t worry, we’re very nearly there with the PR - Mariusz is moving a few of the internal bits exactly where they need to go, and then we’re not far off. Another week or so.

Thanks for your earlier input here. Having folks give it a run adds confidence.

You keep safe too. What a time hey? :slightly_smiling_face:

1 Like