# Profile JSON field -\> Python **URL:** \ **Category:** Using Django\ **Created:** [September 17, 2020, 4:33pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390 "2020-09-17T16:33:03Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 4:33pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/1 "2020-09-17T16:33:03Z")
I have a query which returns up to 200k rows with a Postgres JSON field. The query logic itself has been optimized, but I suspect that performance can be further improved. I am looking to measure how much time is spent parsing the JSON into Python. How would I go about profiling that? ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [September 17, 2020, 4:42pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/2 "2020-09-17T16:42:37Z")
The easiest way would be to use the shell to manually load the data, and then use the [Python timeit](https://docs.python.org/3.8/library/timeit.html) module to run some experiments. Ken ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 4:45pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/3 "2020-09-17T16:45:01Z")
Is there a way to drill down into how much time is spent querying the database vs the time spent parsing JSON? Would I need to subclass the field and measure the from\_python method? ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [September 17, 2020, 4:47pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/4 "2020-09-17T16:47:11Z")
I’d take a first stab in a different direction. I’d use the values clause to return the data as data rather than as Python objects and factor that out of the overall timings. ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 4:47pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/5 "2020-09-17T16:47:42Z")
Cool, thanks for the idea. ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [September 17, 2020, 4:49pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/6 "2020-09-17T16:49:40Z")
You could also go the raw query route where you time the sql itself rather than any layers in the ORM. Either way, once you retrieve the data, you can then run separate timings to see how quickly they’re deserialized into object instances. ---
**Author:** ![adamchainz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/adamchainz/32/26_2.png) [@adamchainz](https://forum.djangoproject.com/u/adamchainz)\ **Post date:** [September 17, 2020, 5:25pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/7 "2020-09-17T17:25:37Z")
For production data timing, you can also install an APM solution (Scout, New Relic, etc.) that constantly tracks performance of queries across your site, so you always have data. ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 5:38pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/8 "2020-09-17T17:38:56Z")
We have that in place, the slow query logs are surfaced from Aruara to DataDog and AWS QuerySite. ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 5:42pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/9 "2020-09-17T17:42:45Z")
@KenWhitesell Don’t `values` and `values_list` both parse the JSON into Python, just not as an ORM instance? I don’t have to call `json.loads` on a `values_list`. ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [September 17, 2020, 5:57pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/10 "2020-09-17T17:57:37Z")
Could be. We’ve left my scope of knowledge now, everything beyond this is conjecture on my part. I seem to remember that it’s psycopg2 that is doing the raw conversion rather than anything in the ORM. (From the [psycopg2 docs on JSON](https://www.psycopg.org/docs/extras.html?highlight=json#json-adaptation): Reading from the database, `json` and `jsonb` values will be automatically converted to Python objects.) It goes on to say that you can bypass this process by casting the column to a `text` field. So if you’re needing to do more fine-grained analysis of the timing, you might want to run the raw queries with and without casting the column. (You could then also run the raw query through psql to factor out the rest of the overhead introduced by psycopg2 and python.) If the total amount of data is large enough to warrant concern, you might even want to evaluate the network overhead involved if you’re not running this on the same system as the database. (And if you are on the same system, you might want to look at the difference between using a tcp port through the loopback address vs a unix socket if you’re not already using one.) Ken ---
**Author:** ![sww314](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/sww314/32/1942_2.png) [@sww314](https://forum.djangoproject.com/u/sww314)\ **Post date:** [September 17, 2020, 8:32pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/11 "2020-09-17T20:32:11Z")
I would attempt to determine where the big problem is in the database or not. In the past, I have found these articles helpful to try and figure that out: > **[Improve Serialization Performance in Django Rest Framework](https://hakibenita.com/django-rest-framework-slow)** > > How we reduced serialization time by 99%! > **[Web API performance: profiling Django REST framework](https://www.dabapps.com/blog/api-performance-profiling-django-rest-framework/)** > > Blog Post by Tom Christie In some recent work, using `queryset.filter().values('field1')` results in big improvements for me. This selects less data from the database. ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 8:50pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/12 "2020-09-17T20:50:42Z")
@KenWhitesell I now see [the docs](https://docs.djangoproject.com/en/3.1/ref/models/fields/#django.db.models.Field.from_db_value) are explicit too that the conversion happens at a lower level than Django. I executed the query directly against the `psycopg2` client with json (the default) and again with the json casted as a string (`json_field::text`). Without the json -\> python conversion, the query is ~3x faster than with the python conversion. Thanks for your help. I’m going to look into different serialization options that might have better performance. ---
**Author:** ![avremel](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/avremel/32/2085_2.png) [@avremel](https://forum.djangoproject.com/u/avremel)\ **Post date:** [September 17, 2020, 8:54pm UTC](https://forum.djangoproject.com/t/profile-json-field-python/4390/13 "2020-09-17T20:54:49Z")
Yep, selecting only what I need with `values_list`, and am specific within the JSON field (`field__key`) as well. Thanks for the links.