I’ve working on some refactoring and one thing that was changes in some model is handling statuses.
Previously statuses used varchar field with choices, after refactoring, new table was added with list of all statuses + l18n to those status names and some other stuff.
What was important is to keep old data and migrate to a different type. PostgreSQL have useful tool in altering column type called USING expression, however that was not available via Django AlterField, so instead of running Django code I run a few RawSQLs:
Renaming status column to status_id
ALTER COLUMN TYPE + USING to change text to associated status id
SET DEFAULT to status id 1 instead of text
Add foreign key constraint (the name for fk I copied from other fk names generated by django and only CRC part of that name was not particularly set to anything meaningful)
All of those have reverse SQL’s as well and they work perfectly, but now I have problem on Django side, whenever I try to create new makemigration it try to alter this column and if I run it’s migration then crashes because it couldn’t find old status column (column does not exist), so how can I tell Django that it already migrated?
Other error
Also apparently there is other error, previously I used db_default on column normally, but IDE doesn’t like db_default on ForeignKey type (Pylance error: No overloads for "__new__" match the provided arguments), but this may be separate issue not related to the topic and I could suppress it with # type: ignore, although that wouldn’t be perfect.
The problem is when I call manage.py makemigrations it create migration for that column that I already migrated and I can’t suppress that action, while the coulmn is already renamed, changed type and get foreign key.
class Foo(models.Model):
status = models.CharField(choices=STATUSES, db_default="initial_status")
Then in my case I had to make it into something like:
class Foo_status(models.Model):
name = models.CharField(choices=STATUSES)
# other fields
class Foo(models.Model):
status = models.ForeignKey(Foo_status, on_delete=models.RESTRICT, db_default=1)
So I made migration for that kind of operation manually via RawSQL as I explained in 1st post
One important note is that before column name was status in DB, but now it become status_id (django automatically add suffix)
It frist look at models.py and then look at db, in db all the migrations are recorded.(not the table itself, but in a different table).
If a migration is identical in models.py and db. It won’t run it again.
But if django can’t see a record about migration, it may try to run it.
That’s not a problem. Because foringnkey fields are id fields.
2) part
I had same kind of problems before. This happens because old migration files contain the old fields. Then django try to remove it. But it’s not there.
So, solution would be to remove all migrations (frist make a backup copy, then delete). And then run migrations.
I had same kind of problems before. This happens because old migration files contain the old fields. Then django try to remove it. But it’s not there.
So, solution would be to remove all migrations (frist make a backup copy, then delete). And then run migrations.
There is no old migrations besides prior migrations which cannot be deleted (they are already on production), and because I have git in repository I can see which files are new in IDE, so there is no leftover files. Removing old migration is not an option.
I can’t personally find out where does Django keep those old fields states hidden in, but my manual migrations should put an equal state for the field I migrated to with the only exception is hash in foreign key name.
But even if I keep those migration changes now it just try to work with status field which is already become status_id field (as with other foreign keys). So I cannot scratch my head around this fact, that there is no way to tell Django that it already migrated. It’s already renamed, it’s already in type of PK from other table, it is populated with values and FK have identical properties. What does not like Django I cannot really understand.
And I use that SQL alternative to migrate manually:
migrations.RunSQL("""ALTER TABLE api_foo RENAME status TO status_id;""",
"""ALTER TABLE api_foo RENAME status_id TO status;"""),
migrations.RunSQL(
f"""
ALTER TABLE api_foo
ALTER COLUMN status_id
TYPE bigint
USING (
CASE status_id
{" ".join(f"WHEN {sn!r} THEN {sid}" for sid, sn in STATUS_REL.items())}
ELSE 1
END
);
""",
f"""
ALTER TABLE api_foo
ALTER COLUMN status_id
TYPE character varying
USING (
CASE status_id
{" ".join(f"WHEN {sid} THEN {sn!r}" for sid, sn in STATUS_REL.items())}
ELSE 'initial_status'
END
);
"""),
migrations.RunSQL("""ALTER TABLE ONLY api_foo ALTER COLUMN status_id SET DEFAULT 1;""",
"""ALTER TABLE ONLY api_foo ALTER COLUMN status_id SET DEFAULT 'initial_status';"""),
migrations.RunSQL(
"""
ALTER TABLE api_foo
ADD CONSTRAINT api_foo_status_id_00000000_fk_api_status_id FOREIGN KEY (status_id)
REFERENCES public.api_foo_statuses (id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE RESTRICT
DEFERRABLE INITIALLY DEFERRED;
""",
"""
ALTER TABLE api_foo
DROP CONSTRAINT api_foo_status_id_00000000_fk_api_status_id;
"""),
No, there are a bunch of migration files in .py format. And the last file is migration that I did which caused me troubles specified in this exact topic.
Ok then, but all previous migrations that was made are already in production, I cannot delete them. I only created new one, very last one, and it is what causing me problems when I create migrations after it.
I mentioned example code for my migration of this field, it should do the same what Django do, except I cannot run Django code because it will erase my data on production, which is why I made those custom SQLs.
I still can redo last migration since it is not on production yet, but I cannot touch previous ones. If there is something missed then I would like to know.
The SQL’s in post above transform my column in exact state that I would like it to be, so they are correct and correct in both ways (I already tried to reverse migration to confirm, that they work), the only problem that was remained is to tell Django about this, that I don’t need to migrate this field anymore.
Problem is django remember the old db state, and when django look at old state and new state, they are different, so, django try to create a migration.