[GSOC 2024] Auto-importing in the shell

Django Internals Mentorship
1

Hi everyone!
I’m very entusiast that my proposal for GSOC’24 has been accepted! Thanks to @DevilsAutumn for feedback about a possible implementation of this feature and for helping me in writing the proposal.

This topic will be the one where I will update the progress of the PR: so every feedback is welcome. I’m very excited and entusiastic about this project and I’m looking forward to implement the auto-import for the shell.

Possible useful links:

In the coming days I will create a new PR that is going to be the one where the major part of the work will be done. In the mean time, if anyone wants to say something about this approach or the implementation described in the proposal, I’m here to listen all your feedback and I’ll be happy to review my work with your help.

Thanks,
Salvo Polizzi

4 Likes
2

Congrats @salvo-polizzi ! I’m really excited to work with you.
Just to be clear on what our final Shell will look like, I want to note down all the functionalities of shell_plus that we plan to add in Django Shell here. This way we can get to know the views of the community as well.

  1. By default, we’ll not print all model names in the Shell but it would be possible with a flag.
  2. Adding SHELL_PLUS_IMPORTS settings, see: shell_plus — django-extensions 3.2.3 documentation
  3. Adding SHELL_PLUS_PRINT_SQL settings, see: shell_plus — django-extensions 3.2.3 documentation
  4. Import all models using their full paths and then import the models in the order of the app names, starting with the first encountered app.
  5. User should be able to subclass the Shell command and override a method.
  6. By default, Django supports IPython, BPython, and standard shell runners. I believe we should enhance the current shell to allow users to add additional shell runners (jupyter notebook, ptpython etc) while preserving the auto-import functionality across all runners.
1 Like
3

Hi @salvo-polizzi , congratulations on being accepted. I proposed this project last year and I’m glad to see it is being picked up. I am not your mentor because I failed to log onto the GSoC panel, but I’m on hand to review your PRs.

Please review the past discussion with another potential candidate: Have Solved 80% of auto_importing Shell feature , and their PR: [gsoc] django_auto_import shell feature by Ammar-Munirr · Pull Request #17826 · django/django · GitHub .

In terms of features, I propose we don’t directly copy shell_plus. Instead, we should try to build a minimal, extensible design.

django-extensions’ design has “grown by committee” (or anarchy). Pretty much all PRs were merged, meaning its features are pretty haphazard. Its code shows that internally, with untested blocks, “secret” undocumented features, and general messiness. We’d rather make fewer solid features in Django, so we can maintain them for a long time.

Here’s how I’d refine Bhuvnesh’s suggestions:

  1. The shell command gains a new method, called, say, get_namespace(). This method returns a dictionary of names to objects, which are all added into the namespace.

  2. The default implementation of get_namespace() adds all models from apps.get_models(). We can have some debate about what to do with name collisions. But whatever we do, models in earlier apps need to take precedence, as that’s the precedence order used elsewhere, such as for management command lookup.

  3. The SHELL_PLUS_IMPORTS is not needed. Instead, we document how to extend get_namespace() to add other imports, something like:

# myapp/management/commands/shell.py
from django.core.management.commands import shell

class Command(shell.Command):
    def get_namespace(self):
        from django.core.urls import resolve, reverse
        
        return {
            **super().get_namespace(),
            "resolve": resolve,
            "reverse": reverse,
        }

  1. No feature to “display all imported objects”, nor an option to turn it on. I think we should stick with a simple message like “X objects automatically imported.”. Users can use globals() or shell-specific features to see all imported objects, if and when they care.

  2. Copying the functionality from SHELL_PLUS_PRINT_SQL is out of scope.

  3. This functionality should work in the supported shells as much as possible.

  4. There’s no need to add support for extending with other shells, as that already exists. One can override the command with a subclass and add an extra method with its name in the shells attribute. But this is undocumented, so it would be a nice bonus to make a side PR documenting how to use this feature.

Thanks again for working on this project, I’m excited to see how it goes!

3 Likes
4

Thanks @adamchainz for your valuable feedback, it clears also my mind about what to do. I also want to inform the community that I’ve created a draft PR where all the work will be done.

5

One other point @DevilsAutumn messaged me on Discord: we should make sure the default imports are loaded for -c / --command as well, which bypasses launching a shell interface by using exec():

The globals() argument can be expanded. That would allow quick one liners like:

$ ./manage.py shell -c 'print(User.objects.count())'
9001
2 Likes
6

Oh, yeah, we should also make sure the default imports are loaded for the stdin feature too, the block below the -c one.

1 Like
7

shell_display
shell_display944×130 21 KB

This is how the import of the objects is currently displayed, following the advice of @adamchainz. What do you think?

8

Hi folks!

I’m here with a little update about the work we are currently doing.

There’s an ongoing discussion about how to handle name collisions. The approach we are considering is to give precedence to models in the earlier apps listed in INSTALLED_APPS. Additionally, we are thinking of providing a way to access overridden models. @adamchainz suggested importing all model modules as <app_label>_models and then accessing these overridden models by <app_label>_models.Model, for example, auth_models.User.

If anyone has any ideas or would like to suggest a strategy to adopt for handling name collisions, I’d be happy to hear them.

9

Hi, One thing that I still feel would be better to add is the SHELL_IMPORTS settings. I see the the shell customization on 2 levels.

At level 1, users would just want to import extra modules, submodules, methods etc. And I think this will be the most common use case.
At level 2, users would want to perform some kind of manipulation or do some pre-processing on their data. This one would be comparatively uncommon.

At this point, to achieve both levels, overriding get_namespace is the only option. But with SHELL_IMPORTS settings, we can super simplify things for atleast users at level 1(where most of the users lie?). As a developer, I would not want to create a bunch of files and folders, subclass a command and override a method just to import few methods/submodules/modules for testing. Rather I would prefer to write 3-4 lines in the settings.py and expect django to handle the complications of importing them.

Yes, If I had some complex operations to perform before launching the shell or want to use some other python runner, then I would definitely have no problem to subclass the shell command and override a method.

Would be happy to know the opinions of the community on this.

1 Like
10

For those who are curious of this feature, please see:

Testing and feedback is always welcome :+1:

3 Likes