If the issue is a real performance issue based upon measurements of real problems and not just the identification of an N+1 situation, then the solution is to restructure your data to avoid using GFKs. (And given that the Admin should not be your primary interface from a user-facing perspective, the N+1 issue in the Admin really shouldn’t be a huge problem.)
See Avoid Django's GenericForeignKey - lukeplant.me.uk and Generic relation vs Many to Many for a couple of perspectives on this. (The first article in particular identifies some alternatives.)