Possible migration file missing from third party package?

I was about to package my Django app and publish the first version on my server.
Build my docker container and run my compose-file.

Migration error

When the app was starting I got a migration error:

curated-app       | Traceback (most recent call last):
curated-app       |   File "/code/manage.py", line 22, in <module>
curated-app       |     main()
curated-app       |   File "/code/manage.py", line 18, in main
curated-app       |     execute_from_command_line(sys.argv)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
curated-app       |     utility.execute()
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
curated-app       |     self.fetch_command(subcommand).run_from_argv(self.argv)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 414, in run_from_argv
curated-app       |     self.execute(*args, **cmd_options)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 460, in execute
curated-app       |     output = self.handle(*args, **options)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 98, in wrapped
curated-app       |     res = handle_func(*args, **kwargs)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 108, in handle
curated-app       |     executor = MigrationExecutor(connection, self.migration_progress_callback)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
curated-app       |     self.loader = MigrationLoader(self.connection)
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
curated-app       |     self.build_graph()
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/loader.py", line 276, in build_graph
curated-app       |     self.graph.validate_consistency()
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 198, in validate_consistency
curated-app       |     [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 198, in <listcomp>
curated-app       |     [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
curated-app       |   File "/usr/local/lib/python3.10/site-packages/django/db/migrations/graph.py", line 60, in raise_error
curated-app       |     raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
curated-app       | django.db.migrations.exceptions.NodeNotFoundError: Migration self_hosted_apps.0005_apppage_logo_svg dependencies reference nonexistent parent node ('wagtailsvg', '0005_alter_svg_file')

So I checked for the migration files which had these mentioned:
wagtailsvg', '0005_alter_svg_file.
I have removed all migrations and created a new one, but still the same problem.

I am using the package wagtailsvg · PyPI
My understanding now is that a migration from that package itself is missing but I am not sure on that part.
I looked on the repo and the 5th migration file is not available:
wagtailsvg/wagtailsvg/migrations at master · Aleksi44/wagtailsvg · GitHub

One of my not working migrations.

# Generated by Django 4.0.6 on 2022-07-30 20:49

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('wagtailsvg', '0005_alter_svg_file'),
        ('self_hosted_apps', '0005_apppage_logo_svg'),
    ]

    operations = [
        migrations.RemoveField(
            model_name='apppage',
            name='date',
        ),
        migrations.AlterField(
            model_name='apppage',
            name='logo_svg',
            field=models.ForeignKey(blank=True, help_text='SVG', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='wagtailsvg.svg'),
        ),
    ]

The odd thing is. My local setup does not have that problem.
But the database is also older than the newly created production setup.

Questions

Is my assumption correct?
If not what do I need to do to solve the problem?

Do a manage.py showmigrations. Does it show a migration by that name of having been executed? If so, you’ll need to remove that entry from the table. (Deleting migration files after they’ve been applied is frequently a bad thing.)

As a debugging / diagnostic step, I’d try deleting all migrations that depend on that 0005_ migration and all migrations that depend on them. Start with a fresh (empty) database, do a migrate, then a makemigrations / migrate to pick up everything new - just to verify that something isn’t more fundamentally broken.

Then, depending upon the degree of brokeness, you could try emptying the migrations table in your real database and apply the existing migrations with the --fake parameter to register with Django that those migrations had been previously applied.

Good think nothing is live right now. So I can delete tables and databases as I want.

I did another test just now

I have deleted my postgres db and removed all my migrations for the affected app.
I then start up the postgres again.
Then I start the python process.
Create all migrations

python manage.py makemigrations           
Migrations for 'wagtailsvg':
  /home/andrej/.pyenv/versions/curated/lib/python3.10/site-packages/wagtailsvg/migrations/0005_alter_svg_file.py
    - Alter field file on svg
Migrations for 'self_hosted_apps':
  self_hosted_apps/migrations/0001_initial.py
    - Create model AppListPage
    - Create model AppPage

As you can see this will create a migration file in the package layer.

When I migrate everything works fine.

python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, django_celery_results, generic_page, home, self_hosted_apps, sessions, taggit, wagtailadmin, wagtailcore, wagtaildocs, wagtailembeds, wagtailforms, wagtailimages, wagtailredirects, wagtailsearch, wagtailsvg, wagtailusers
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying django_celery_results.0001_initial... OK
  Applying django_celery_results.0002_add_task_name_args_kwargs... OK
  Applying django_celery_results.0003_auto_20181106_1101... OK
  Applying django_celery_results.0004_auto_20190516_0412... OK
  Applying django_celery_results.0005_taskresult_worker... OK
  Applying django_celery_results.0006_taskresult_date_created... OK
  Applying django_celery_results.0007_remove_taskresult_hidden... OK
  Applying django_celery_results.0008_chordcounter... OK
  Applying django_celery_results.0009_groupresult... OK
  Applying django_celery_results.0010_remove_duplicate_indices... OK
  Applying django_celery_results.0011_taskresult_periodic_task_name... OK
  Applying wagtailcore.0001_squashed_0016_change_page_url_path_to_text_field... OK
  Applying wagtailcore.0017_change_edit_page_permission_description... OK
  Applying wagtailcore.0018_pagerevision_submitted_for_moderation_index... OK
  Applying wagtailcore.0019_verbose_names_cleanup... OK
  Applying wagtailcore.0020_add_index_on_page_first_published_at... OK
  Applying wagtailcore.0021_capitalizeverbose... OK
  Applying wagtailcore.0022_add_site_name... OK
  Applying wagtailcore.0023_alter_page_revision_on_delete_behaviour... OK
  Applying wagtailcore.0024_collection... OK
  Applying wagtailcore.0025_collection_initial_data... OK
  Applying wagtailcore.0026_group_collection_permission... OK
  Applying wagtailcore.0027_fix_collection_path_collation... OK
  Applying wagtailcore.0024_alter_page_content_type_on_delete_behaviour... OK
  Applying wagtailcore.0028_merge... OK
  Applying wagtailcore.0029_unicode_slugfield_dj19... OK
  Applying wagtailcore.0030_index_on_pagerevision_created_at... OK
  Applying wagtailcore.0031_add_page_view_restriction_types... OK
  Applying wagtailcore.0032_add_bulk_delete_page_permission... OK
  Applying wagtailcore.0033_remove_golive_expiry_help_text... OK
  Applying wagtailcore.0034_page_live_revision... OK
  Applying wagtailcore.0035_page_last_published_at... OK
  Applying wagtailcore.0036_populate_page_last_published_at... OK
  Applying wagtailcore.0037_set_page_owner_editable... OK
  Applying wagtailcore.0038_make_first_published_at_editable... OK
  Applying wagtailcore.0039_collectionviewrestriction... OK
  Applying wagtailcore.0040_page_draft_title... OK
  Applying wagtailcore.0041_group_collection_permissions_verbose_name_plural... OK
  Applying wagtailcore.0042_index_on_pagerevision_approved_go_live_at... OK
  Applying wagtailcore.0043_lock_fields... OK
  Applying wagtailcore.0044_add_unlock_grouppagepermission... OK
  Applying wagtailcore.0045_assign_unlock_grouppagepermission... OK
  Applying wagtailcore.0046_site_name_remove_null... OK
  Applying wagtailcore.0047_add_workflow_models... OK
  Applying wagtailcore.0048_add_default_workflows... OK
  Applying wagtailcore.0049_taskstate_finished_by... OK
  Applying wagtailcore.0050_workflow_rejected_to_needs_changes... OK
  Applying wagtailcore.0051_taskstate_comment... OK
  Applying wagtailcore.0052_pagelogentry... OK
  Applying wagtailcore.0053_locale_model... OK
  Applying wagtailcore.0054_initial_locale... OK
  Applying wagtailcore.0055_page_locale_fields... OK
  Applying wagtailcore.0056_page_locale_fields_populate... OK
  Applying wagtailcore.0057_page_locale_fields_notnull... OK
  Applying wagtailcore.0058_page_alias_of... OK
  Applying wagtailcore.0059_apply_collection_ordering... OK
  Applying wagtailcore.0060_fix_workflow_unique_constraint... OK
  Applying wagtailcore.0061_change_promote_tab_helpt_text_and_verbose_names... OK
  Applying wagtailcore.0062_comment_models_and_pagesubscription... OK
  Applying wagtailcore.0063_modellogentry... OK
  Applying wagtailcore.0064_log_timestamp_indexes... OK
  Applying wagtailcore.0065_log_entry_uuid... OK
  Applying wagtailcore.0066_collection_management_permissions... OK
  Applying wagtailcore.0067_alter_pagerevision_content_json... OK
  Applying wagtailcore.0068_log_entry_empty_object... OK
  Applying wagtailcore.0069_log_entry_jsonfield... OK
  Applying generic_page.0001_initial... OK
  Applying home.0001_initial... OK
  Applying home.0002_remove_homepage_body... OK
  Applying home.0003_homepage_body... OK
  Applying taggit.0001_initial... OK
  Applying taggit.0002_auto_20150616_2121... OK
  Applying taggit.0003_taggeditem_add_unique_index... OK
  Applying wagtailsvg.0001_initial... OK
  Applying wagtailsvg.0002_svg_edit_code... OK
  Applying wagtailsvg.0003_alter_svg_options... OK
  Applying wagtailsvg.0004_remove_svg_edit_code... OK
  Applying wagtailsvg.0005_alter_svg_file... OK
  Applying wagtailimages.0001_squashed_0021... OK
  Applying wagtailimages.0022_uploadedimage... OK
  Applying wagtailimages.0023_add_choose_permissions... OK
  Applying wagtailimages.0024_index_image_file_hash... OK
  Applying self_hosted_apps.0001_initial... OK
  Applying sessions.0001_initial... OK
  Applying taggit.0004_alter_taggeditem_content_type_alter_taggeditem_tag... OK
  Applying wagtailadmin.0001_create_admin_access_permissions... OK
  Applying wagtailadmin.0002_admin... OK
  Applying wagtailadmin.0003_admin_managed... OK
  Applying wagtaildocs.0001_initial... OK
  Applying wagtaildocs.0002_initial_data... OK
  Applying wagtaildocs.0003_add_verbose_names... OK
  Applying wagtaildocs.0004_capitalizeverbose... OK
  Applying wagtaildocs.0005_document_collection... OK
  Applying wagtaildocs.0006_copy_document_permissions_to_collections... OK
  Applying wagtaildocs.0005_alter_uploaded_by_user_on_delete_action... OK
  Applying wagtaildocs.0007_merge... OK
  Applying wagtaildocs.0008_document_file_size... OK
  Applying wagtaildocs.0009_document_verbose_name_plural... OK
  Applying wagtaildocs.0010_document_file_hash... OK
  Applying wagtaildocs.0011_add_choose_permissions... OK
  Applying wagtaildocs.0012_uploadeddocument... OK
  Applying wagtailembeds.0001_initial... OK
  Applying wagtailembeds.0002_add_verbose_names... OK
  Applying wagtailembeds.0003_capitalizeverbose... OK
  Applying wagtailembeds.0004_embed_verbose_name_plural... OK
  Applying wagtailembeds.0005_specify_thumbnail_url_max_length... OK
  Applying wagtailembeds.0006_add_embed_hash... OK
  Applying wagtailembeds.0007_populate_hash... OK
  Applying wagtailembeds.0008_allow_long_urls... OK
  Applying wagtailembeds.0009_embed_cache_until... OK
  Applying wagtailforms.0001_initial... OK
  Applying wagtailforms.0002_add_verbose_names... OK
  Applying wagtailforms.0003_capitalizeverbose... OK
  Applying wagtailforms.0004_add_verbose_name_plural... OK
  Applying wagtailforms.0005_alter_formsubmission_form_data... OK
  Applying wagtailredirects.0001_initial... OK
  Applying wagtailredirects.0002_add_verbose_names... OK
  Applying wagtailredirects.0003_make_site_field_editable... OK
  Applying wagtailredirects.0004_set_unique_on_path_and_site... OK
  Applying wagtailredirects.0005_capitalizeverbose... OK
  Applying wagtailredirects.0006_redirect_increase_max_length... OK
  Applying wagtailredirects.0007_add_autocreate_fields... OK
  Applying wagtailredirects.0008_add_verbose_name_plural... OK
  Applying wagtailsearch.0001_initial... OK
  Applying wagtailsearch.0002_add_verbose_names... OK
  Applying wagtailsearch.0003_remove_editors_pick... OK
  Applying wagtailsearch.0004_querydailyhits_verbose_name_plural... OK
  Applying wagtailsearch.0005_create_indexentry... OK
  Applying wagtailsearch.0006_customise_indexentry... OK
  Applying wagtailusers.0001_initial... OK
  Applying wagtailusers.0002_add_verbose_name_on_userprofile... OK
  Applying wagtailusers.0003_add_verbose_names... OK
  Applying wagtailusers.0004_capitalizeverbose... OK
  Applying wagtailusers.0005_make_related_name_wagtail_specific... OK
  Applying wagtailusers.0006_userprofile_prefered_language... OK
  Applying wagtailusers.0007_userprofile_current_time_zone... OK
  Applying wagtailusers.0008_userprofile_avatar... OK
  Applying wagtailusers.0009_userprofile_verbose_name_plural... OK
  Applying wagtailusers.0010_userprofile_updated_comments_notifications... OK

The problem is, that I don’t know how to package that package migration in my docker container.
Unless I do a makemigration when I build the container. This seems like the wrong way.
But I could be mistaken on that opinion.

Is there a way to create that migration and put it somewhere else than in the package scope?

What is that migration that was created in the package?

My guess is that something changed within Django such that a default that applied in an earlier release has been changed, so this package needs to alter a table to reflect the new settings.

It is this migration which is created locally for the wagtailsvg package:

/home/andrej/.pyenv/versions/curated/lib/python3.10/site-packages/wagtailsvg/migrations/0005_alter_svg_file.py

The content of that file:

# Generated by Django 4.0.6 on 2022-08-02 21:20

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('wagtailsvg', '0004_remove_svg_edit_code'),
    ]

    operations = [
        migrations.AlterField(
            model_name='svg',
            name='file',
            field=models.FileField(upload_to='svg', verbose_name='file'),
        ),
    ]

I see that this migration is not part of the packege on github:

I am also not sure if it should be part of the package.

I also contacted the developer but I think we are still not sure how to solve this.

I see the following in the models.py definition for that model:
upload_to=getattr(settings, 'WAGTAILSVG_UPLOAD_FOLDER', 'media'),

Do you have that variable set in your settings? That appears to be what’s causing this migration to be generated. If you remove that setting, this migration shouldn’t be generated.

1 Like

Indeed I have setup

WAGTAILSVG_UPLOAD_FOLDER = 'svg'

I followed the instructions form the README:

Set the SVG download folder in the Django settings

WAGTAILSVG_UPLOAD_FOLDER = 'svg'
Default value is ‘media’

I have deleted all migrations , db and the local package.
When creating migrations again only migrations for my app will be created and not for the local wagtailsvg package:

python manage.py makemigrations
Migrations for 'self_hosted_apps':
  self_hosted_apps/migrations/0001_initial.py
    - Create model AppListPage
    - Create model AppPage

This works after a quick test.

Bummer I can not change that folder without making it hard to package the app into docker.

I’m not sure (in fact, I’m quite doubtful) that this migration is really “necessary” from a database perspective. I think you could, in development, allow that setting to remain the default. Create your migrations as normal. Then, when you’re ready to dockerize it, add that setting. I think that would work the way you’re looking to use it.