Hi all ![]()
I’d like feedback on an optimization before writing a patch.
**The problem.** `MigrationRecorder.has_table()` checks whether `django_migrations` exists by listing every table the connection can see (`introspection.table_names()`). On PostgreSQL that’s `get_table_list()`: a sequential scan of all of `pg_class`, with `pg_table_is_visible()` evaluated for every table. And a single `migrate` does it **3 times**. `_has_table` is cached per recorder, but the loader (`build_graph`), `check_consistent_history`, and the executor each create their own `MigrationRecorder`.
For a normal project that’s instant. With a large catalog it isn’t. We run a PostgreSQL database with 5.2M `pg_class` rows (705k tables, from many schemas). There, one call took **1.3–3.5 s** and about 3M buffer hits, and it grew the backend’s catalog cache to **352 MB**. Across parallel migrate processes, that memory growth exhausted the database server during a deployment. It was the top query of the whole run, above every DDL statement. Heavily partitioned databases would hit the same thing.
**What I’m proposing.** Check for the one table directly:
```python
# base introspection: default keeps today’s behavior
def table_exists(self, cursor, table_name):
return table_name in self.table_names(cursor)
# PostgreSQL: index lookup, same search_path semantics as pg_table_is_visible()
def table_exists(self, cursor, table_name):
cursor.execute(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class c "
"WHERE c.oid = to_regclass(%s) AND c.relkind IN ('f', 'm', 'p', 'r', 'v'))",
\[self.connection.ops.quote_name(table_name)\],
)
return cursor.fetchone()\[0\]
```
Then `has_table()` calls `introspection.table_exists(…)`. On the catalog above that’s about 0.1 ms instead of seconds. We run the equivalent as a local patch and saw DB load during migrate drop about 80%.
**Questions:**
1. Is a new (possibly private) introspection method like `table_exists()` acceptable, or would you rather keep this inside `MigrationRecorder` with a backend check?
2. Would it also be worth having one `migrate` share a single recorder (or cache the result on the connection), so the check runs once instead of 3 times? Or is that more churn than it’s worth once the check itself is cheap?
Thanks!