server and shell using different python versions

Using Django
1

The Django REPL is running python 3.8. I know that because when I run python3 manage.py shell, I can do this:

In [1]: import sys

In [2]: sys.version
Out[2]: '3.8.2 (default, Mar 26 2020, 15:53:00) \n[GCC 7.3.0]'

But the server is using 3.7. I know that because when I load this view:

def write_python_version ( request ):
  return render (
    request,
    'run_make/write_python_version.html',
    { "version" : str ( sys.version ) } )

with this template:

The Python version is:
  {{version}}<br>

the page reads:

The Python version is: 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]

That seems bad on its face, but I would ignore it, if it weren’t for the fact that I think it’s preventing me from importing Pandas. When I try to import it, in Debug Mode, the page shows the following NumPy error. (I never actually tried to import NumPy directly, but I believe Pandas is trying to do so.)

numpy:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/usr/bin/python3"
  * The NumPy version is: "1.20.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
2

How are you running the shell? Do you have a virtual environment active? What do you get if you just run a python3 -V? How about python -V?

How are you running the server?

3

Thanks for responding, Ken!

I run the shell with python3 manage.py shell, from the project root. I start the server with service apache2 start.

Python 3.8 is provided by Conda, while 3.7 is not:

appuser@127:/mnt/django$ which python3
/opt/conda/bin/python3
appuser@127:/$ which python3.7
/usr/bin/python3.7

I’m using the Docker image described here. That doesn’t (explicitly?) install python3, but it does RUN apt install -y python3-pip. Maybe that installs a competing version of Python?

It looks like Conda + Pip can lead to trouble. But I need some packages that aren’t packaged for Conda, and I don’t want to have to package them myself. I hope that’s not the only way forward.

4

What do you want your “official” python installation to be? The Conda environment, the system environment (discouraged), or a virtual environment?

If the Conda-based environment is your preferred environment, then you want to ensure that your apache configuration is using it for its python runtime, not the system default installation.

Adding docker to the mix does add another level of complexity. If you’re building everything on the Conda packages, then you probably want to create your container using that as well.

But I think the key item here is that you want to make sure you’re consistent from top to bottom. Build one virtual environment for your system and ensure that that’s what you use in all situations.

(We ditched Conda a couple years ago, outside some very specific and limited applications. Its quirks were more than what we wanted to deal with and we never faced the problems it directly addresses as being problems for us.)

5

I’d like Conda to be the default. In my apache2.conf file I’ve got the following line:

WSGIDaemonProcess www-data python-path=/mnt/django:/opt/conda/lib/python3.8/site-packages/

which doesn’t even say to look in /usr/bin, the place Apache is finding Python 3.7. I’ve looked around online and can’t find anything else that says how to tell Apache which Python to use. Do you know?

Adding docker to the mix does add another level of complexity. If you’re building everything on the Conda packages, then you probably want to create your container using that as well.

I don’t understand. All my code runs in the Docker container. When I build the image for that container, I install Conda first, and every package that can be conda-installed. Then I use pip3 to install a few things that aren’t packaged for Conda.

Build one virtual environment for your system and ensure that that’s what you use in all situations.

I think that’s what I’m doing, by doing everything inside the docker container. It’s just that the docker container’s got two versions of python3.

6

When I enter the container as root and as it to apt remove python3.7, I’m told that

The following packages were automatically installed and are no longer required:
  gir1.2-glib-2.0 iptables libexpat1-dev libgirepository-1.0-1 libip4tc0 libip6tc0 libiptc0
  libmpdec2 libnetfilter-conntrack3 libnfnetlink0 libnftables0 libnftnl11 libpython3-dev
  libpython3-stdlib libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
  nftables python-pip-whl python3-minimal python3.7-minimal
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  dh-python libapache2-mod-wsgi-py3 python3 python3-asn1crypto python3-cffi-backend
  python3-crypto python3-cryptography python3-dbus python3-dev python3-distutils
  python3-entrypoints python3-gi python3-keyring python3-keyrings.alt python3-lib2to3
  python3-pip python3-pkg-resources python3-secretstorage python3-setuptools python3-six
  python3-wheel python3-xdg python3.7 python3.7-dev ufw

So maybe it’s that a lot of the things I installed via apt caused python3.7 to be installed? I don’t recognize a lot of them, but libapache2-mod-wsgi-py3 and python3-pip are crucial.

7

It’s the python-home directive.

I’m assuming you already have this link, but for reference: WSGIDaemonProcess — mod_wsgi 5.0.0 documentation

From the section on python-home:

Note that the Python virtual environment must have been created using the same base Python version as was used to compile the mod_wsgi module. You can’t use this to force mod_wsgi to somehow use a different Python version than it was compiled for. If you want to use a different version of Python, you will need to reinstall mod_wsgi, compiling it for the version you want.

Adding the Conda libraries just to python-path is going to create a mix of versions between libraries and the python runtime, and anything with a compiled component is likely to have problems in that environment.

8

Thanks, Ken. I determined that installing any of ufw, mod-wsgi, apache or pip via apt causes Python 3.7 to be installed, so I installed them all by hand. It seemed to work, but when I try to start Apache it complains that it can’t find libpcre.so.1. But libpcre is in fact installed.

I had forgotten how hard it was to set up Apache in the first place. To avoid another week of playing whackamole I’m planning to just let the version conflict persist, and use the csv library instead of pandas from Django. I don’t (from Django) need anything fancy that would actually require Pandas. I do from the microsimulation that Django runs, but it uses a syscall to start another python environment which has access to it.

9

I can understand where using mod_python may be the easiest way of doing it - as long as you have a rather “vanilla” installation. But once you get outside the norm, things can get hairy rather quickly.

We had the situation where different apps (non-Django) were using different versions of Python, running under the same server at the same time.

I’m reasonably certain that that is why we originally adopted using the mod_wsgi / uwsgi pair rather than mod_python. (We also found it useful to be able to manage those different applications independently rather than having them all tied to the apache instance.)

Getting used to working with uwsgi also made the transition to nginx rather trivial.

10

Hey Jeffery, in case you haven’t 100% given up, did you happen to try any of the approaches in the link beow? In particular, update-alternatives might be helpful since you’re apparently running your app in a Debian-based distribution.