postgres app + psycopg2 not providing postgis functionality?

Using Django
1

i’m upgrading postgres on MacOS using the
Postgres app. it says it includes
PostGIS.

I have used pip to load django and psycopg2-binary within a virtualenv, and
in settings.py i have specfied: 'ENGINE': 'django.db.backends.postgresql_psycopg2' .

if i use the python3 manage.py dbshell command within that
virtualenv, i get into the new psql version (v.13.3).

But when i run some code that used to work (w/ Postgres 9.5),
everything works fine in
django.db.models.sql.compiler.SQLCompiler.setup_query() until…

it gets to a Point field. At that point it calls
django.contrib.gis.db.models.fields.GeometryField.select_format()
and throws this exception:
AttributeError: 'DatabaseOperations' object has no attribute 'select'

Using pdb to poke around, i find that the compiler parameter
passed to select_format() seems to have the correct value for
connection.Database = <module 'psycopg2' from '.../virtualenv/dj3/lib/python3.8/site-packages/psycopg2/__init__.py'>

Thanks for any suggestions as to what the issue might be, or how to find out more.

2

Did you create the extension within your database?

(create extension postgis; from the psql shell, or as a command executed from within code)

3

yes, it’s already there:

oakcrime=# CREATE EXTENSION postgis;
ERROR: extension “postgis” already exists

4

I think the next step then is to examine the view or other code that you’re running, as well as other environmental differences that may exist.

What are the old and new versions of each of Python, Django, psycopg2, and PostGIS? Are there any other key libraries involved that may have changed?

5

First Ken, thanks much for helping me puzzle this out. The big change i anticipated was postgres 9.5->13.2. Before:
postgres 9.5, PostGIS=??, Django==1.11.28, psycopg2-binary==2.7.6.1
Now:
postgres 13.2, PostGIS=3.1, Django==3.2.3, psycopg2-binary==2.8.6
nothing else exotic being used in the ORM layers. (Did you relabel this post as “Using Django” vs. the “ORM-related” i had used?)

Within the view, the offending query is also pretty straight-forward:

    	qs = OakCrime.objects.filter(opd_rd = cid)
		alreadyPresent = qs.exists()
		if alreadyPresent:
			matchObjList = list(qs)
6

Huge changes between Django 1.11 and 3.2, the other changes you identified probably don’t affect anything. Also, what versions of Python being used for each?

There’s not enough of your code posted to draw the connection between the error you listed and the snippet you posted here. While we may not need to see the entire view and models, it’s really tough to try and diagnose this without seeing a lot more detail. (Also, it’s likely to be helpful to see the complete traceback. Sometimes the real problem is located somewhere other than the location where the exception is thrown.)

Yes. From the Categories page:

Django Internals
For discussing the development of Django itself, like the django-developers mailing list.

So the Internals / ORM category is for discussing changes and fixes to the ORM itself, not for discussions about how to use the ORM.

For those conversations, they belong here:

Using Django
For discussions about using Django, building sites and projects, like the django-users mailing list.

7

Re: post labeling: got it!

python2.7 before, python3.8.5 now. So yes, python, django AND postgres have moved considerably!

full traceback:

Traceback (most recent call last):
File “…/virtualenv/dj3/ea_opal/manage.py”, line 22, in
main()
File “…/virtualenv/dj3/ea_opal/manage.py”, line 18, in main
execute_from_command_line(sys.argv)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/core/management/init.py”, line 419, in execute_from_command_line
utility.execute()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/core/management/init.py”, line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/core/management/base.py”, line 354, in run_from_argv
self.execute(*args, **cmd_options)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/core/management/base.py”, line 398, in execute
output = self.handle(*args, **options)
File “…/virtualenv/dj3/ea_opal/dailyIncid/management/commands/harvestSocrata.py”, line 603, in handle
harvest(startDate)
File “…/virtualenv/dj3/ea_opal/dailyIncid/management/commands/harvestSocrata.py”, line 576, in harvest
summRpt = mergeList(results,srcLbl,verboseFreq=‘chgSocDT’,rptAll=True) # verboseFreq=100,
File “…/virtualenv/dj3/ea_opal/dailyIncid/management/commands/harvestSocrata.py”, line 449, in mergeList
matchObjList = list(qs)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/query.py”, line 262, in len
self._fetch_all()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/query.py”, line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/query.py”, line 51, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 1162, in execute_sql
sql, params = self.as_sql()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 513, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 55, in pre_sql_setup
self.setup_query()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 46, in setup_query
self.select, self.klass_info, self.annotation_col_map = self.get_select()
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 267, in get_select
sql, params = col.select_format(self, sql, params)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/db/models/expressions.py”, line 388, in select_format
return self.output_field.select_format(compiler, sql, params)
File “…/virtualenv/dj3/lib/python3.8/site-packages/django/contrib/gis/db/models/fields.py”, line 280, in select_format
return compiler.connection.ops.select % sql, params
AttributeError: ‘DatabaseOperations’ object has no attribute ‘select’

8

Ok, with all the changes in Python between 2.7 and 3, the only recommendation I can suggest is a complete review of your code to ensure that everything that has been changed from Python 2 to Python 3 has been addressed.

Since Django 1.11 is the last to support Python 2, I’d suggest keeping your code on Django 1.11 and doing the conversion to Python 3 under Django 1.11. Once you have your system working on Python 3, you can try going directly to Django 3.2, but you may find it easier to go to Django 2.2 first.

9

So i gather there is no way to inspect into contrib.gis and/or models.sql to get a better sense of what/where is failing?

i think i’m going to come up with a minimal example required to create this fail. And/or trying to reproduce this error in a Linux environment, to see if it’s a Mac issue. i’m less concerned with what used to work than in making it work in the current world. Thanks again for your thoughts.

10

Sure, you could use the debugger and trace your way through the calls. The source for those libraries are definitely available.

However, my inclination is that there isn’t a problem with either of those libraries. They are far too widely used to start by assuming they are the problem.

The “lowest line” in the traceback that’s in your code that I can see is this:

This implies to me that the queryset at that point is improperly constructed - and it’s quite possibly due to a Python 2 → Python 3 data structure issue. My second guess is that it’s Django 1.11 → Django 3.2 difference.

When faced with two such significant changes, I’m always looking to try and narrow down the variables, and since I can’t go to Django 3.2 under Python 2.7, my only choice would be to try it under Django 1.11 with Python 3.

11

i have traced this through, and in the for loop (line#260 of compiler), it happily walks thru all the earlier fields in the data object until it hits point which is a GIS PointField:

0,(Col(dailyIncid_oakcrime, dailyIncid.OakCrime.idx), None)
1,(Col(dailyIncid_oakcrime, dailyIncid.OakCrime.opd_rd), None)
...
10,(Col(dailyIncid_oakcrime, dailyIncid.OakCrime.xlng), None)
11,(Col(dailyIncid_oakcrime, dailyIncid.OakCrime.ylat), None)
12,(Col(dailyIncid_oakcrime, dailyIncid.OakCrime.point), None)

and it fails within contrib.gis.db.models.fields.GeometryField.select_format() .