I haven’t seen this brought up anywhere looking through the forum or the django-developers ML.
The short version is I want to use pytest’s --stepwise for working on the test suite, and it doesn’t seem that pytest works directly out of the box. But maybe it’s only a small wrapper script away from doing it! So if anyone had a setup that they are using for this, I would love to hear it.
I do understand that runtests.py does things already so it’s a bit hard to insist on having support that is checked in. I’m just asking aloud whether someone has something off the source tree they might be using.
I haven’t done this myself, but I think this would be a nice improvement to the test suite, at least in theory. I think the runtests.py approach is a bit too special and would prefer we could run pytest or python -m unittest and get the tests to run properly.
That said, this is probably not a trivial piece of work, so it’ll need someone determined to make it happen.
Yeah… I’m going to try getting it working but I am a bit uncomfortable about upstreaming it, because then it’s like “do we need feature parity on pytest and unittest runners?”
I don’t know what other dev’s feelings are on pytest, but in general it’s likely a good thing that unittest test cases are how Django’s writing tests internally (the world would be a much more complicated place without django.test.TestCase). But at the same time pytest as a test runner is so much nicer and easier to configure IME.
But I’ll see what a source tree that runs pytest looks like
make sure to sort the test suite so that TestCases run before TransactionTestCases. This is done by Django’s current runner and is necessary in particular to deal with ContentType tests. Several tests wipe ContentTypes entirely, and others expect them to be present from the initial migrate call. This was a particularly annoying thing to debug, but fortunately I had looked at the Django test runner recently enough to remember ordering.
call a handful of test utils in a pytest conftest.py at the start of the session
rename a bunch of mixins from things like TestDefault to DefaultTestsMixin, so Pytest doesn’t think they’re tests
disable pytest’s capturing of warnings and pytest-django (if you have it installed)
Skip a pickling test (Pytest’s unittest wrapper isn’t picklable)
One thing I could not port over was the current runner’s “installed application filtering” trick. The current runner looks at the tests you are about to run and tries to filter down test modules to only consider those. Importantly, it does this before importing anything, but runtests.py is handling the “which files am I going to run tests on”.
Pytest has ways of manipulating test and file collection, but the basic idea with pytest is you should be able to import your tests without errors. So while in theory I should be able to do something like “collect all the tests, then filter down, then determine INSTALLED_APPS”, in practice Pytest complains on test imports due to the dreaded “no app_label and not in INSTALLED_APPS” error.
Similarly, there are some skipIfs or the like that expect connection to be set up already, so while the canonical thing for Pytest would be “configure, collect, then run initial setup”, I’m finding myself needing to setup django before collecting tests.
I might be able to resolve both issues, though, hopefully with a minimal amount of trickery. But I do kind of wish that we could somehow stop having code that is just outright not importable sometimes without having to do a prelude.
Otherwise I’m not seeing many issues. You do need a test prelude and teardown for this test suite somewhere. Pytest provides canonical ways of doing that with conftest.py, I’m not sure about unittest.
@KenWhitesell I just noticed the move over to “Using Django”, given that this is about running Django’s test suite (and not running a test suite with Django), and I would like to upstream at least some of the changes I have here, would it not make sense in “Django Internals”?