Deploying with uwsgi and venv

Using Django
1

I’m trying to deploy a django project with the hoster uberspace. They provide you with access to a linux server where you are one of many users. I was able to succesfully get django running without a virtual environment. Now I’m trying with a virtual environment, but I didn’t have succes as of yet. Here is my uwsgi configuration.

[uwsgi]
project = test_project
uid = devspiff
gid = devspiff
base = /home/devspiff

chdir = %(base)/dj_prj/test_project_venv/test_project/test_project
home = %(base)/dj_prj/test_project_venv/
module = test_project.wsgi:application
pythonpath=%(base)/dj_prj/test_project_venv/test_project/test_project/

http = :8000
master = true
processes = 5
# plugin = python

wsgi-file = %(base)/dj_prj/test_project_venv/test_project/test_project/wsgi.py
touch-reload = %(wsgi-file)
static-map=/static=%(base)/dj_stc/test_project/
app = wsgi

Here’s the content of the uwsgi log.

*** Starting uWSGI 2.0.19.1 (64bit) on [Fri Aug 14 03:24:04 2020] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-39) on 09 August 2020 17:57:54
os: Linux-3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020
nodename: bebhionn.uberspace.de
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 20
current working directory: /
detected binary path: /home/devspiff/.local/bin/uwsgi
your processes number limit is 103763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
*** starting uWSGI Emperor ***
Python version: 3.8.3 (default, May 14 2020, 12:59:20)  [GCC 8.3.1 20190311 (Red Hat 8.3.1-3)]
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from test_project.ini
[uwsgi-static] added mapping for /static => /home/devspiff/dj_stc/test_project/
*** Starting uWSGI 2.0.19.1 (64bit) on [Fri Aug 14 03:24:04 2020] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-39) on 09 August 2020 17:57:54
os: Linux-3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020
nodename: bebhionn.uberspace.de
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 20
current working directory: /home/devspiff/uwsgi/apps-enabled
detected binary path: /home/devspiff/.local/bin/uwsgi
chdir() to /home/devspiff/dj_prj/test_project_venv/test_project/test_project
your processes number limit is 103763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
building mime-types dictionary from file /etc/mime.types...1060 entry found
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :8000 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:40730 (port auto-assigned) fd 3
Python version: 3.8.3 (default, May 14 2020, 12:59:20)  [GCC 8.3.1 20190311 (Red Hat 8.3.1-3)]
PEP 405 virtualenv detected: /home/devspiff/dj_prj/test_project_venv/
Set PythonHome to /home/devspiff/dj_prj/test_project_venv/
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1eefed0
your mercy for graceful operations on workers is 60 seconds
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 20143)
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xe90ab0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437520 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
added /home/devspiff/dj_prj/test_project_venv/test_project/test_project/ to pythonpath.
ModuleNotFoundError: No module named 'test_project'
unable to load app 0 (mountpoint='') (callable not found or import error)
Traceback (most recent call last):
  File "/home/devspiff/dj_prj/test_project_venv/test_project/test_project/wsgi.py", line 12, in <module>
	from django.core.wsgi import get_wsgi_application
ModuleNotFoundError: No module named 'django'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 20145)
Fri Aug 14 03:24:04 2020 - [emperor] vassal test_project.ini has been spawned
spawned uWSGI worker 1 (pid: 20146, cores: 1)
spawned uWSGI worker 2 (pid: 20147, cores: 1)
spawned uWSGI worker 3 (pid: 20148, cores: 1)
spawned uWSGI worker 4 (pid: 20149, cores: 1)
spawned uWSGI worker 5 (pid: 20150, cores: 1)
spawned uWSGI http 1 (pid: 20151)
Fri Aug 14 03:24:04 2020 - [emperor] vassal test_project.ini is ready to accept requests

So the above mentioned Python 3.8.3 is the Python version that I used to create the venv. With my hoster in the shell, I have to use commands like “python3.8 -m venv /path/to/new/virtual/environment” as the standard python command is linked to Python 2.7.5. Now here is a strange behaviour I noticed. I used pip3.8 to install django. Django is now installed with python3.8. When I open a python3.8 shell and import django, it’s fine. When I open a python shell (2.7.5.) and import django, it is not installed. When I run “pip install django” (2.7.5) it says the requirement is met already.
Googling the error message that I got, I found a stackoverflow thread that suggested, that the Python version of uwsgi and the one of the venv have to be the same.


That makes no sense to me. I tried adding the path to the virtual environment to my pythonpath, but it did not help. Any suggestions would be appreciated. My knowledge about linux is very limited. It seems that this might be an access rights problem, but I’m in over my head.
2

Check out the virtualenv parameter. I set it to the base directory of the virtualenv and it seems to work fine for me. Oh, I see you’re using it under its name home - try taking off the trailing slash. Same with your pythonpath parameter.

Ken

3

So, I think the trailing slashes were part of the problem, although I could swear they were there in the examples that I read through.
But there was a second problem. On my providers shell, the comman python will use python 2.7.5., python3.8 will use python 3.8.3, and so on. Inside the venv it’s different. python3.8 still uses python 3.8.3, however python overrides the python command. When I use python within the venv, it will use python 3.8.3. The same goes for pip. So when I was using pip3.8 inside the venv it installed packages globally and not into the venv. That is why django was not found.
Linux can be quite hard when it comes to rights and contexts and what commands do depending on them.

4

I’m taking this to mean that you had activated the venv and pip was still installing packages globally? (If the venv is active, you should just be able to use pip and not pip3.8. I guess it’s possible that even if the venv is active, if it’s not using the venv version of pip, it could cause problems.)

If you don’t mind, would you post the output of which pip, and which pip3.8, for both when the venv is active and when it’s not? (It’s just to satisfy my own curiosity.) Actually, based on your comment, it might also be worth seeing the output from which python and which python3.8 both with the venv active and not active.

Ken

5

Since you say there “was” a second problem, does everything seem to be working now, or are you still having trouble? :slight_smile:

Regarding the directory paths you describe:

chdir = %(base)/dj_prj/test_project_venv/test_project/test_project
pythonpath=%(base)/dj_prj/test_project_venv/test_project/test_project/

Am I correct in guessing that these point to an “inner” ‘test_project’ folder, ie the folder where you have your ‘settings.py’ file? (rather than the “outer” ‘test_project’ folder, where ‘manage.py’ resides) I’m guessing this since you also have wsgi-file = %(base)/dj_prj/test_project_venv/test_project/test_project/wsgi.py in there. If so, are you certain that that is what uwsgi wants?

I’m in a similar process of deploying a django application to a VPS (using Linode and following a tutorial). In my case, I’m using Apache with mod_wsgi, but there seem to be many similarities between mod_wsgi and uwsgi. When I set the python path, I use a path ‘/home/datalowe/project_folder’, where ‘project_folder’ is the project root folder, ie it holds ‘manage.py’ et c. After reading about your problems I took a look at uWSGI’s quickstart guide. They have this snippet:

# ...
chdir = /home/foobar/myproject/
wsgi-file = myproject/wsgi.py
#

They never describe explicitly exactly which folder (“outer” or “inner” project directory) they mean by /home/foobar/myproject, but since they for the wsgi-file seem to use a relative path (i. e. relative to the chdir path) of ‘myproject/wsgi.py’, I would assume that just like mod_wsgi, uwsgi expects ‘chdir’ to be set to the outer path.

A side note is that the quickstart guide I linked above seems to indicate that the pythonpath variable should really only be necessary for old Django versions - I don’t entirely understand why/how this works though.

Is your venv (or whatever you chose to call it) folder inside of your “outer” ‘test_project’ folder, right alongside ‘manage.py’? If so, I think that’s another strong reason for setting the chdir path to your “outer” directory.

On my providers shell, the comman python will use python 2.7.5., python3.8 will use python 3.8.3, and so on. Inside the venv it’s different. python3.8 still uses python 3.8.3, however python overrides the python command.

This is what should/usually happens when you use a virtual environment, AFAIK. You should notice the same thing if you use a virtual environment locally, in your own computer’s terminal, regardless of operating system. I know this for sure when it comes to Mac OSX and Linux, but I would assume the same holds for Windows. So you don’t need to consider it a problem, if you can embrace it then it really simplifies things. Instead of having to specify python3 each time or even more specific python/pip versions, you can just rely on the virtual environment to consistently use the versions that you (implicitly or explicitly) specified when creating the virtual environment.

So when I was using pip3.8 inside the venv it installed packages globally and not into the venv. That is why django was not found.

(@KenWhitesell) Actually, based on your comment, it might also be worth seeing the output from which python and which python3.8 both with the venv active and not active.

Of course, our setups aren’t identical. (I’m using Ubuntu 20.04, Python 3.8.2, pip 20.0.2, and the “regular” venv package). But if I activate the virtual environment first…

datalowe@servername-server:~/myproject$ source .venv/bin/activate

(I’ve named my venv folder .venv, to make it hidden)
… so that I get this type of prompt…

(.venv) datalowe@servername-server:~/myproject$

… I get the following:

(.venv) datalowe@servername-server:~/myproject$ which pip
# /home/datalowe/myproject/.venv/bin/pip
(.venv) datalowe@servername-server:~/myproject$ which pip3.8
# /home/datalowe/myproject/.venv/bin/pip3.8

If I try installing something while explicitly specifying pip3.8…

(.venv) datalowe@servername-server:~/myproject$ pip3.8 install numpy

… I can then, as expected, find the following folders in ~/myproject/.venv/lib/python3.8/site-packages/:

  • numpy
  • numpy-1.19.1.dist-info
  • numpy.libs

If I then try, within the virtual environment (so still with the (.venv) part at the beginning of the prompt), to enter the python shell (python), and then >>> import numpy, I can successfully import the package. If I exit the python shell, exit (deactivate) the virtual environemtn and then enter the “regular” python3 shell and try >>> import numpy again, then I get a ModuleNotFoundError, as expected. Did you do anything different when installing your packages, @Spidiffpaffpuff?

6

@datalowe
No, it’s working fine now. Today I uploaded a second project. The test_projetct runs on port 8000, the other one runs on 8010. I can access both projects and they are working fine. I’m happy now. :slightly_smiling_face:

As for the pythons and pips:

  • “which pip” from venv
    ~/dj_prj/test_project_venv/bin/pip

  • “which pip3.8” from venv
    ~/dj_prj/test_project_venv/bin/pip3.8

  • “which pip” without venv
    ~/.local/bin/pip

  • “which pip3.8” without venv
    ~/.local/bin/pip3.8

  • “which python” from venv
    ~/dj_prj/test_project_venv/bin/python

  • “which python3.8” from venv
    ~/dj_prj/test_project_venv/bin/python3.8

  • “which python” without venv
    /usr/bin/python

  • “which python3.8” without venv
    /usr/bin/python3.8

If I then try, within the virtual environment (so still with the (.venv) part at the beginning of the prompt), to enter the python shell ( python ), and then >>> import numpy , I can successfully import the package. If I exit the python shell, exit ( deactivate ) the virtual environemtn and then enter the “regular” python3 shell and try >>> import numpy again, then I get a ModuleNotFoundError , as expected. Did you do anything different when installing your packages, @Spidiffpaffpuff?

Well I have the odd experience that I would install from within the venv using pip3.8, enter a shell from within the venv and the just installed packages could not be imported.

7

Great! Then I guess my post was more in-depth than it needed to be, but at least it was a good review exercise for me :wink:

Really very odd. If it happens again then it might be worth going to your venv folder’s site-packages subdirectory to see if the packages were actually installed within the virtual env or not.