Making MigrationRecorder.has_table() cheap on large PostgreSQL catalogs

Hi all :waving_hand:

I’d like feedback on an optimization before writing a patch.

**The problem.** `MigrationRecorder.has_table()` checks whether `django_migrations` exists by listing every table the connection can see (`introspection.table_names()`). On PostgreSQL that’s `get_table_list()`: a sequential scan of all of `pg_class`, with `pg_table_is_visible()` evaluated for every table. And a single `migrate` does it **3 times**. `_has_table` is cached per recorder, but the loader (`build_graph`), `check_consistent_history`, and the executor each create their own `MigrationRecorder`.

For a normal project that’s instant. With a large catalog it isn’t. We run a PostgreSQL database with 5.2M `pg_class` rows (705k tables, from many schemas). There, one call took **1.3–3.5 s** and about 3M buffer hits, and it grew the backend’s catalog cache to **352 MB**. Across parallel migrate processes, that memory growth exhausted the database server during a deployment. It was the top query of the whole run, above every DDL statement. Heavily partitioned databases would hit the same thing.

**What I’m proposing.** Check for the one table directly:

```python

# base introspection: default keeps today’s behavior

def table_exists(self, cursor, table_name):

return table_name in self.table_names(cursor)

# PostgreSQL: index lookup, same search_path semantics as pg_table_is_visible()

def table_exists(self, cursor, table_name):

cursor.execute(

    "SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class c "

    "WHERE c.oid = to_regclass(%s) AND c.relkind IN ('f', 'm', 'p', 'r', 'v'))",

    \[self.connection.ops.quote_name(table_name)\],

)

return cursor.fetchone()\[0\]

```

Then `has_table()` calls `introspection.table_exists(…)`. On the catalog above that’s about 0.1 ms instead of seconds. We run the equivalent as a local patch and saw DB load during migrate drop about 80%.

**Questions:**

1. Is a new (possibly private) introspection method like `table_exists()` acceptable, or would you rather keep this inside `MigrationRecorder` with a backend check?

2. Would it also be worth having one `migrate` share a single recorder (or cache the result on the connection), so the check runs once instead of 3 times? Or is that more churn than it’s worth once the check itself is cheap?

Ticket: #37379 (MigrationRecorder.has_table() lists every table in the database, 3 times per migrate) – Django

Thanks!