Adding ptpython to the manage.py shell command

Before creating the ticket, and since the auto importing shell is being worked on as a part of the GSoC, I think some discussion is needed.

Ptpython is an REPL library.
Based on Github stars, I’d say it’s popular enough to be added, and it has more stars than the already supported BPython (2.6k stars, vs 5.1k for Ptpython) The other supported one, IPython is at 16.2k stars.

The main implementation would be identical to the bpython one:

   def ptpython(self, options):
        import ptpython

        ptpython.embed()

However, there was already an attempt at adding it 9 years ago and closed a wontfix.
There were even mentions of deprecating the shell command

I think in an ideal world manage.py shell would not exist, but it is widely used and documented, so people are probably right that pragmatically speaking it’s not worth deprecating it at this point. We could just leave [ipython/bpython support] until it breaks and deprecate it then rather than continuing to fix/update it.

Since the shell command is actually being extended as part of the GSoC, could there be a change on the position of not adding new REPLs?

2 Likes

If you have ptpython installed in the active virtual environment, will it be used if you run a normal python shell and a django shell? I’m fairly sure that’s how ipython works today. If that’s how ptpython works, then I’m not sure of the reason to add it here.

There’s a list of supported shells that can be used. They will be used by default if they exist.

Otherwise they won’t be imported by default.

There’s the PYTHONSTARTUP env variable to indicate a file run at startup, which can be used to set the default shell. Ipython is not set as the default python shell without this.

Setting the shell in that as suggested in the ptpython docs while for example having ipython installed has a weird behaviour. First it loads the Ipython shell, and then loads the contents of the env file. So you end up with Ptpython loaded inside an Ipython shell.

The command has a --no-startup option to ignore the file, but it only works with the basic python shell.

1 Like

Thank you for the clarification! I think this change makes sense to me. pypython has broad support across operating systems and it’s been around for a long time (it supports python 2.6!). If we’re looking to keep shell around, then I think we can add ptpython to it.

I’m not sure quite how widely it’s used, but I do like ptpython. Going by stars and how much I’ve seen them both, it’s probably more common than bpython by now, which we do support. Since the deprecation was also closed wontfix, I think we should be looking at making sure we’re supporting well-used REPLs.

I’m in favour of adding ptpython. Maybe @salvo-polizzi would like to do this as a side quest whilst working on the GSoC project? Seems like we have the short method to add and would need a test, documentation, release note, and a manual test.

1 Like

I don’t want to give them extra work, this can be added after that project is done since the autoimport of modules may be different for each shell. I’ll be happy to help if we want to do it now.

Regarding tests, I see that there are tests to error if the shell library is not installed, however there is not a case to tests that check the success of the import, would that make sense to add or would it be too much of a pain, due to having to install the dependency?

1 Like

I think we can do it now - if you want to open the PR, then feel free.

We have a PR to add test coverage: Refs #35517 -- Improved test coverage of shell command. by salvo-polizzi · Pull Request #18265 · django/django · GitHub . This uses mocks though, to avoid the test suite requiring all the different dependencies being installed.

Hi @alexgmin, are you going to work on this new feature? I think that if ptpython is integrated, it should support the auto-importing feature. @adamchainz, what do you think? However, I’ll be happy to help if needed.

I was planning create the bug and work on it later this week.
In the PR that adds it, do you want me to create the tests with the same mocks you used in Refs #35517 -- Improved test coverage of shell command. by salvo-polizzi · Pull Request #18265 · django/django · GitHub or do you prefer to do it in that PR?

I’ve just marked that PR as “ready for checkin”, so hopefully a fellow will merge it soon and you’ll be able to add a new ptpython test in your new PR.

Yes, definitely!

Same here, tag us on your PR @alexgmin.

The test PR is now merged, so you can base your work on it @alexgmin , if you have the time.

I’m against adding support for any “Xpython” (ptpython or any other). Keeping support for bpython and ipython has caused some maintenance burden in the past and we decided to not fix it again and keep bpython and ipython only until they brake again.

I’d rather deprecate and remove ipython and bpython support.

I’ve looked at the git blame and Trac and the only issue I found was that iPython changed the way to embed it in 1.0. Django seemed to add support before that version using a non-stable API.

Could you expand on what has been the maintenance burden?

All new options introduce the maintenance burden. We decided to not check and confirm support for the new versions of ipython and bpython, and simple remove them when it will be broken. I don’t see any point in adding the “official” support for any other shell. It doesn’t need to live in Django itself, if you want to use any new custom interpreter, you can do this in two lines, e.g.

$ ptpython
>>> import django
>>> django.setup()
....

I see where ptpython is supported by the django-extensions package within their shell_plus command.

Personally, I would have no problem with deprecating bpython and ipython support from within Django itself, in favor of recommending people use django-extensions for enhanced shell functionality.

If that’s the stance, then I’m not sure the work of this PR adding test coverage via mocks to the other two shells will be that useful.

This is correct right now, however after the GSoC project, it won’t be since it would be missing the auto imported models.

Since I see that there’s not enough support for adding it, I think this topic has reached it’s resolution unless someone wants to deprecate the support for the library shells right now.

Every integration comes with a maintenance burden. We can’t just reject ideas because they have some cost; we have to weigh the cost against the benefit.

To me, an alternative shell is a great benefit. It offers a better developer experience and can especially help those starting with Django from a beginner or different context (IPython, for example, is very familiar to those in data science).

I think Django should err on the side of “generosity” here, and include more integrations for stable, popular shells like ptpython. I hope that we go ahead here - it seems there’s broad support in the replies to this thread.

I also don’t think the maintenance cost is particularly large.

First, these are the tiniest of methods—just two lines: an import and a function call. If the function call needs updating, it’s really not much work. We can probably rely on a user who likes that integration to submit a PR. (For example, I will be happy to write fixes for any change to IPython’s API.)

Second, change seems to be rare. As so far noted, IPython’s API only changed once, when reaching its 1.0. And ptypthon.embed() has been stable for at least 10 years since this commit.


django-extensions is a bit of a “code dumping ground” after spending its early years merging submitted PRs for random ideas that people came up with. There are features in its code without any documentation or tests. As such, I prefer not to use it nor recommend it.

I would rather we bring the best ideas into Django core, where it’s easier for everyone to use them out of the box - the “batteries included” approach. That’s why I proposed the “auto-importing shell” GSoC project and am happy to mentor Salvo on it.


Can you link to this conversation?

1 Like

The question is how many Django developers need this, 10%?, 5%?, maybe less. Django should be the core framework, there is no need to support every possible options. You can always create a 3rd party package for that :wink:

I’d argue with that. If we officially support another shell we will have to deal with issues like, Django shell crash on my Windows machine, on my ARM64, on my niche operating system, etc. Removing support for anything with our policy is a long-term process.

Will try to find it, but as a starting point, have you read the discussion in #19737? We were close to completely removing shell command. TBH, I sympathize with this idea.

I agree with @felixxm here I use the shell command as a Django Developer but I don’t know what ptpython is.

If I knew what ptpython is and I wanted to use it I could use it the way you suggested by calling django.setup or maybe if there’s a need it can start as a third party package and if it gains enough popularity that a major percentage of django users use it then we can add.

Although if this is something which can be added with less amount of work then I agree to it’s addition.

I am just a normal Django user providing my opinion.