# GSOC Parallel Test Runner **URL:** \ **Category:** Google Season of Code\ **Created:** [March 1, 2020, 5:56pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334 "2020-03-01T17:56:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Valz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/valz/32/701_2.png) [@Valz](https://forum.djangoproject.com/u/Valz)\ **Post date:** [March 1, 2020, 5:56pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/1 "2020-03-01T17:56:34Z")
Hello everyone, My name is Ahmad. I’m a Computer Engineering and Science sophomore at the German University in Cairo. I read through the GSOC Idea list and the parallel test runner stood out for me. It seems to be one of the more interesting project ideas. I’m going to start going through the testing framework tickets on Trac to familiarize myself with it, but I’d definitely love any pointers or advice anyone can throw at me. P.S I seriously love how friendly and open to discussion everyone is, whether on the mailing list or on the forums; I was a bit intimidated at first but that faded away quickly. ---
**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\ **Post date:** [March 3, 2020, 11:11am UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/2 "2020-03-03T11:11:00Z")
Hi Ahmad. Cool! I think this is a great topic. Previously we’ve been able to run the test suite in parallel on Linux and macOS. With Python 3.8, the default (`multiprocessing`) start method on macOS went from `fork` to `spawn`, because `fork` is deprecated and will be removed in a future version of macOS. For now, you can work around it, by setting it back to `fork`, but this means we we’re down to just Linux for parallel test execution (on Python 3.8+) It would be good to get it going with the `spawn` method, so we can run at full speed on macOS and Windows too. On macOS with Python 3.8+ or Windows (with any supported Python) run the django test suite, a project’s tests with `./manage.py test` using the `--parallel=4` (say) flag. You’ll see lots of errors (most are `AppRegistry` issues.) The trick will be to work out how to adjust `django/test/runner.py::_init_worker()` to set the right data. It’s a nice project because you get to dig right into the multiprocessing module: what exactly is the difference between `fork` and `spawn`? (OK, “environment and file descriptors are not shared…” — but what does _that_ mean? 🙂) Then there’s working out how to get the right DB connections to each worker, and… And then there’s testing across platforms. And… I’d first of all review the original ticket and discussions. Did this issue come up when it was first implemented? (Windows support would surely have been mentioned…) Hopefully that gets you started. 🙂 Kind Regards, Carlton ---
**Author:** ![ichlaffterlalu](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/ichlaffterlalu/32/812_2.png) [@ichlaffterlalu](https://forum.djangoproject.com/u/ichlaffterlalu)\ **Post date:** [March 10, 2020, 5:06pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/3 "2020-03-10T17:06:45Z")
Hello, Carlton. My name is Ichlasul **Affan**. I’m a first year Computer Science magister student at University of Indonesia. I’m looking through all GSoC project ideas and interested in fixing parallel testing for Windows, especially because I mostly used Windows as my main work OS when I’m doing Django projects. Firstly, thank you very much for your initial advice. Just like Ahmad, I am currently feeling a bit intimidated on how I should initialize on contributing to Django. I am currently working on 1 issue in Trac but it’s about adding warning on UniqueConstraint for MySQL (easy-picking), to familiarize myself on how I should code. Is it worth it to continue working on that issue? I didn’t found new unreviewed issue on Django Test when I started to pick issues at Trac. Secondly, I have read some of codes on `django/test/runner.py` and `django/core/management/ __init__.py`. It shows that Django loads `Apps` registry at first by executing `django.setup()`, after that it continues by running the specific command (in this case, spawn a DiscoverRunner). Is it okay to start with `django.setup()` on each thread that being spawned using `spawn` method? Please correct me if I’m wrong. 😄 That’s all for now. Any further advice is very appreciated. Thanks. ---
**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\ **Post date:** [March 11, 2020, 1:21pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/4 "2020-03-11T13:21:46Z")
Hi @ichlaffterlalu. Welcome. `setup()` ultimately just sets some state, so maybe we can reuse that, rather than recalling it, but, yes we need the app registry populated in each worker. Equally we need to make sure each worker is using the correct database connection (there’s one cloned for each worker). Part of the project is working out these issues, and how to test them… ---
**Author:** ![Valz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/valz/32/701_2.png) [@Valz](https://forum.djangoproject.com/u/Valz)\ **Post date:** [March 22, 2020, 3:46pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/5 "2020-03-22T15:46:12Z")
@carltongibson I’ve read through documentation and the PR that implemented the parallel test runner itself. The difference between spawn and fork seems to be whether or not there is an inherited state from parent to child processes which in our case means that setting up state has to be done per-process. Wouldn’t each worker use the correct database connection because of get\_test\_db\_clone\_settings? The only situation I see right now where that method fails to work is if the cloning process itself failed and the get\_test\_db\_clone\_settings gives an incorrectly configured connection Is there something I’m missing about the difference between spawn and fork affecting worker initialization? ---
**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\ **Post date:** [March 22, 2020, 5:12pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/6 "2020-03-22T17:12:17Z")
Hi @Valz. Yeah… it’s **just** a case of initialising the workers correctly. (I didn’t look into the DB connection handling as yet…) ---
**Author:** ![Valz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/valz/32/701_2.png) [@Valz](https://forum.djangoproject.com/u/Valz)\ **Post date:** [March 22, 2020, 5:26pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/7 "2020-03-22T17:26:28Z")
Alright, thank you for your reply. I wanted to ask: should I submit my proposal on this forum or on the django-developers mailing list? I was also considering to expand the scope of this proposal to include bringing in the Oracle database into the parallel test runner. This of course would mean a lot more testing but I believe Oracle users would get a huge performance boost when running tests. I’ve already read through the original PR’s discussion about implementing Oracle cloning and how much of a headache that was compared to other databases, but it seems like a feature that has a ton of support. I can of course ask on the django-developers mailing list to confirm this ---
**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\ **Post date:** [March 22, 2020, 5:29pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/8 "2020-03-22T17:29:29Z")
It needs to go to the GSoC portal to be submitted. I would offer to read a draft, but the reality of that with the current situation with covid-19 I really don’t have the capacity. ---
**Author:** ![Valz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/valz/32/701_2.png) [@Valz](https://forum.djangoproject.com/u/Valz)\ **Post date:** [March 22, 2020, 6:19pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/9 "2020-03-22T18:19:21Z")
It’s okay. Hopefully the situation will get better. I’ll submit it on the GSoC portal later then. Is it in line with community guidelines to ask for someone to review my draft on the mailing list or would that be off-topic? ---
**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\ **Post date:** [March 22, 2020, 6:27pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/10 "2020-03-22T18:27:24Z")
There’s no problem is asking. 🙂 ---
**Author:** ![Valz](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/valz/32/701_2.png) [@Valz](https://forum.djangoproject.com/u/Valz)\ **Post date:** [March 22, 2020, 6:28pm UTC](https://forum.djangoproject.com/t/gsoc-parallel-test-runner/1334/11 "2020-03-22T18:28:18Z")
Alright great 🙂 Wanted to have an explicit blessing because I understand django has a ton of structure Thanks again for your replies!