The future of the Watchman autoreloader

Hey all :wave:, long time no see.

I鈥檓 getting back into contributing to Django after a long hiatus, and I figured I鈥檇 take a look at the autoreloader.

The tl;dr is that while Watchman is pretty OK, the pywatchman library seems to be a bit of a dumpster fire. It鈥檚 completely broken (see pywatchman raises SystemError on Python 3.10 路 Issue #970 路 facebook/watchman 路 GitHub) so anyone using 1.4.1 won鈥檛 benefit from any watchman related autoreloading despite watchman itself working fine. With the recent layoffs at Meta, and given that no release has been successfully made since 2017, I think that waiting for a release isn鈥檛 a great idea.

I see that we have two paths to take:

  1. Rip out watchman and support something else
  2. Replace the pywatchman client

I think that option 2 is the most viable because majority of the code in the pywatchman library is around the bser codec they support and some specific stuff around Windows IO.

I think that creating a small client using the JSON encoding scheme would be pretty simple. I鈥檓 going to create a proof of concept to see but I wanted to gather some feedback or suggestions. If the 鈥渂uild a simpler pywatchman client鈥 is viable, there are questions about ownership and where the code should live. I鈥檇 propose a separate library under the Django organisation rather than integrating this code directly into Django.

As a side note, I鈥檓 looking forward to meeting some of you face-to-face in Edinburgh!

2 Likes

I鈥檝e been building pywatchman from source and it鈥檚 working fine. The FB team has ignored packaging it, but they do seem to maintain the Python bindings.

I had a crack at this today and came up with GitHub - orf/watchman-client

I鈥檓 not a fan of the pywatchman library - it seems overly complex and requires us to implement a number of additional things that it really should take care of.

One pretty core example is simply watching directories (roots). Watchman may be watching a parent directory which means watching a root may return a relative root. Unfortunately all events are propagated as if they came from the parent root, which requires us to do extra housekeeping in order to match a relative subscription to a parent root.

The initial implementation was only about 130 lines or so but I went a bit further and implemented this housekeeping and a few convenience methods, so it now weighs in at ~340 lines. I also noticed a few deficiencies in my Django implementation that I fixed here (namely removing the need for sleeping, and there鈥檚 a small memory leak that鈥檚 hidden by the restarting).

Suggesting users install the library from git doesn鈥檛 seem like great advice even if main does currently work.

IMO shipping this ourselves could be a viable solution that wouldn鈥檛 be terribly complex to maintain, and would help the wider ecosystem.

Wow, that looks great. I never bothered installing pywatchman. Given how short it is, I am personally leaning towards adding it to Django itself (or at least adding it as dependency). I really would love to see a autoreload solution that does not drain battery more than needed where I do not have to remember to install something into every venv I want to use Django in (I understand that watchman still needs to be installed, but that is a one time thing).

1 Like

This looks great. I hope to test it soon.

For path 1, supporting watchfiles out of the box would be nice too (which is what uvicorn uses).

I like the approach of using watchfiles. I started a package to support watchfiles: GitHub - adamchainz/django-watchfiles: Use watchfiles in Django鈥檚 autoreloader. . It鈥檚 alpha quality still but enough people have found and tried it.

If @orf or others would like to get it to the same level as the watchman integration, happy to help review and merge. I will hopefully have time to work on it later in the year. That could then be a tested base for adding support into Django.

I would definitely be up for using watchfiles, it looks quite good. Couple of thoughts:

It doesn鈥檛 seem to be possible to add/remove watches at runtime. The common use case for this is watching imported modules in site-packages (which change at runtime), I guess we鈥檇 need to create a single site packages watch and do some manual filtering.

But this does get a bit complex - if you have a virtual environment that is inside the project directory those will overlap. The thing I do like about watchman is that it hides this overlapping complexity from you, but it鈥檚 more than possible to implement ourselves.

Watchman does also have some nice features around delaying notifications during git operations. Again, we can definitely implement this ourselves or possibly add it upstream.

The other complexity is symlinks. I鈥檓 not entirely sure how watchman handles this, other than 鈥淚 think it does and I think it鈥檚 transparent鈥. If it鈥檚 also transparent with watchfiles then I think it鈥檚 a viable replacement.

1 Like

Great research @orf . If you want to play around, I鈥檝e invited you to the GitHub repo as a collaborator.

I鈥檝e been scoping this out a bit and I鈥檝e pushed what I鈥檝e got so far to your project @adamchainz :muscle:

I鈥檝e also got a PR in Django here: Draft: Fixed #34479: Initial attempt at using watchfiles by orf 路 Pull Request #16747 路 django/django 路 GitHub. I鈥檓 seeing some test failures locally around duplicate events and there鈥檚 more work needed around git + default ignores, but it does seem to work and the implementation is pretty simple.

1 Like

Hi @orf, I wanted to ask you about your thoughts on the current state of the autoreloader. I鈥檝e been following this thread loosely as I鈥檝e been working on a related ticket for my company, and it seems that currently there isn鈥檛 a solution for file watching in Django with Python 3.10 outside of django-watchfiles, which is still kind of alpha-ish. I saw your PR is now closed; is there still some work being considered to fix the autoreloader in Django?

@orc contributed to Django-watchfiles. I think the best way forwards is to test it, and if you can, add test coverage. Then we might consider merging it into Django.

1 Like