No, that’s the trick: the assets are not necessarily defined in the model admin class since a ListFilter could be used for mutiples instances of them. The idea is to let the filter define all it need to display. That would allow creating much more complex widgets.
For instance: think of a model that has a creation date and a modification date. You could create a complex widget that lets you select dates on a calendar. But since there are 2 dates on the model, the widget is here twice. So the assets would be sourced by the template twice.
I agree this is not a common use case but this could be useful in some cases and it’s really not hard to implement.