You want your “base model” for your query to be on User then.
You can then annotate the scores for each user.
e.g.
user_list = User.objects.annotate(avg50=..., avg100=...)
If you’re not familiar with annotations, start with the docs at Aggregation | Django documentation | Django. Pay particular attention to the cheat sheet to see some examples, some of which are very close to what you’re looking to do.