You’re really close - you don’t need the Sum function on line two:
CustomUser.objects.annotate(
post_count=Count('posts'),
comment_count=Count('comment'),
total=F('post_count') * 10 + F('comment_count')
).order_by('-post_count', '-comment_count', 'username')
post_count and comment_count are singular values, not a list - there’s nothing to “Sum” here.