Well it looks like calling prefetch_related after union is disallowed which I’m unsure why but if you bundle it up in your first queryset everything should work as expected.
Have a look here.
It looks like we should lift the restriction on using prefetch_related after union as long as the queryset is dealing with model instances (as long as no values / values_list mask has been used).