I have a lot of trouble using dumpdata and loaddata, and I just keep banging my shins when I try to work around it.
Use cases are:
- populating the database with realistic data for tests
- data migrations, moving some data from one system to another
- moving data fixtures from a branch into main
- backup and restore (pg_dump is great for this)
I’d also like something like an import wizard that can handle renaming models and fields, setting default or derived values etc. Django-import-export is good (very good) in situations where we get the same spreadsheet every month and need to upload it, but it isn’t for json.
I’m trying to lay this out analytically but its been a bit of a chaotic search, with me trying things with loaddata, trying to implement natural-keys, trying to write my own scripts and so on:
contenttypes and permissions need to use natural-keys; the pks set in a new blank database won’t necessarily align.
difficulties with data from models with concrete inheritance. If model B inherits from model A, dumpdata will output instances of B as separate object from A. They are united in the data model by having the same pk, but they can’t just be created independently - model B instances need their associated model A data.
difficulties with natural keys for some objects where they are very relational - e.g. we have at least one model which is unique by its relationships through a ManyToMany relationship and exists as a shortcut to that unique set. (If there are 12 feasible combinations of factors [(a, b), (1, 2, 3), (X, Y)] , this table has 12 rows and each row relates to a unique set of factors (1: a1X, 2: b1X etc.) Defining the natural _key function for all the models was OK, but for some of these types the queries to get_or_create() them had some very extended foreign key chains.)
I’ve tried the django-extensions DumpScript and RunScript (but it was a while ago) without success, and when I tried just now to refresh my memory it seems they don’t support Django 4.0 yet (‘can’t import ‘smart_text’ from django.utils.encoding’). Not sure if this was the reason but it does seem to export only app by app, which seems like I could still mess it up.
What are societies conventions on this? Is there a really good tool I should know about?