Queryset Group not working.

Hi,

I am trying to retrieve records over a 30 day period, then group via the fk and annotate the item with a number of different metrics, but for some reason the grouping isn’t grouping. Instead its just returning every item.

im using .values(contract).annotate(...

    end_date = datetime.now().date() - timedelta(days=1)
    start_date = end_date - timedelta(days=29)
    
    metrics = ContractMetric.objects.filter(capture_date__range=[start_date, end_date])

    # Subqueries for latest and previous values
    latest_floor_price_subquery = metrics.filter(contract=OuterRef('contract')).order_by('-capture_date').values('floor_price')[:1]
    previous_floor_price_change_subquery = metrics.filter(contract=OuterRef('contract'), capture_date=start_date).values('floor_price')[:1]
    previous_unique_holders_subquery = metrics.filter(contract=OuterRef('contract'), capture_date=start_date).values('total_owners')[:1]

    contract_sums = metrics.values('contract__id').annotate(
        capture_count=Count('id'),
        total_volume_traded_change_sum=Sum('total_volume_traded_change'),
        latest_floor_price=Coalesce(Subquery(latest_floor_price_subquery, output_field=FloatField()), Value(0, output_field=FloatField())),
        previous_floor_price=Coalesce(Subquery(previous_floor_price_change_subquery, output_field=FloatField()), Value(0, output_field=FloatField())),
        floor_price_change_over_7_days=ExpressionWrapper(
            F('latest_floor_price') - F('previous_floor_price'),
            output_field=FloatField()
        ),

When i print the value of capture_count i am expecting to get 30. One for each record over the 30 day period.

but it returns 1

    for item in contract_sums:
        print(f"Contract ID: {item['contract']}, Capture Count: {item['capture_count']}")

returns:

Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
Contract ID: 340, Capture Count: 1
...

Im not sure if its something weird with my model or I’m misunderstanding how to group, but i use the same syntax on other views and it seems to work perfectly

class Contract(models.Model):
    address = models.CharField(max_length=150, null=True, blank=True, unique=False)
    opensea_collection = models.URLField(max_length=150, null=True, blank=True, unique=True)
    blockchain = models.CharField(max_length=50, blank=True,null=True)
    slug = models.CharField(max_length=150, null=True, blank=True, unique=False)
    nft = models.BooleanField(default=True)
    creator_royalty_percentage = models.FloatField(null=True, blank=True)

    def __str__(self):
        return str(self.address)

class ContractMetric(models.Model):
    contract = models.ForeignKey(Contract, on_delete=models.CASCADE)
    collection_name = models.CharField(max_length=200, null=True, blank=True)
    capture_date = models.DateField(null=False,blank=True)
    ...
    def __str__(self):
        return str(self.contract)

Where am i going wrong?

Thanks

Tom

Could you provide the SQL being generated (try print(str(contract_sums.query))).

I suspect the subquery annotation might have something to do with it. Also I assume total_volume_traded_change is a field of ContractMetric?

Hi Charettes,

Thanks for the reply.

SELECT "backend_contractmetric"."collection_name", "backend_contractmetric"."contract_id", "backend_contractmetric"."total_owners", "backend_contractmetric"."total_supply", "backend_contractmetric"."mint_date", "backend_contractmetric"."unique_holder_percentage", "backend_contractmetric"."creator_royalty_percentage", "backend_contractmetric"."total_volume_traded", "backend_contractmetric"."total_earned", "backend_contractmetric"."block_value", "backend_contractmetric"."market_cap", "backend_contractmetric"."fully_diluted_valuation", "backend_contractmetric"."circulating_supply", "backend_contractmetric"."exchanges", COUNT("backend_contractmetric"."id") AS "capture_count", COALESCE((SELECT U0."floor_price" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) ORDER BY U0."capture_date" DESC LIMIT 1), 0.0) AS "latest_floor_price", (COALESCE((SELECT U0."floor_price" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) ORDER BY U0."capture_date" DESC LIMIT 1), 0.0) - COALESCE((SELECT U0."floor_price" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0)) AS "floor_price_change_over_7_days", COALESCE(SUM(CASE WHEN COALESCE((SELECT U0."floor_price" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0) = 0.0 THEN 0 ELSE "backend_contractmetric"."floor_price_change_percentage" END), 0.0) AS "floor_price_change_over_7_days_percentage", CASE WHEN COALESCE((SELECT U0."total_owners" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0) = 0.0 THEN 0.0 ELSE ("backend_contractmetric"."total_owners" - COALESCE((SELECT U0."total_owners" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0)) END AS "unique_holder_change_over_7_days", CASE WHEN "backend_contractmetric"."total_owners" = 0 THEN 0.0 ELSE ((CASE WHEN COALESCE((SELECT U0."total_owners" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0) = 0.0 THEN 0.0 ELSE ("backend_contractmetric"."total_owners" - COALESCE((SELECT U0."total_owners" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0)) END / "backend_contractmetric"."total_owners") * 100) END AS "unique_holder_change_percentage_over_7_days", COALESCE(SUM(CASE WHEN "backend_contractmetric"."total_volume_traded_change" = 0.0 THEN 0 ELSE "backend_contractmetric"."total_volume_traded_change" END), 0.0) AS "total_volume_change_over_7_days", COALESCE(SUM(CASE WHEN "backend_contractmetric"."total_volume_traded" = 0.0 THEN 0 ELSE "backend_contractmetric"."total_volume_traded_change_percentage" END), 0.0) AS "total_volume_change_percentage_over_7_days", COALESCE(SUM(CASE WHEN "backend_contractmetric"."total_earned" = 0.0 THEN 0 ELSE "backend_contractmetric"."total_earned_change" END), 0.0) AS "total_royalty_change_over_7_days", COALESCE(SUM(CASE WHEN "backend_contractmetric"."total_earned" = 0.0 THEN 0 ELSE "backend_contractmetric"."total_earned_change_percentage" END), 0.0) AS "total_royalty_change_percentage_over_7_days" FROM "backend_contractmetric" WHERE "backend_contractmetric"."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 GROUP BY "backend_contractmetric"."contract_id", 16, COALESCE((SELECT U0."floor_price" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0), 17, COALESCE((SELECT U0."total_owners" FROM "backend_contractmetric" U0 WHERE (U0."capture_date" BETWEEN 2024-06-19 AND 2024-07-18 AND U0."capture_date" = 2024-06-19 AND U0."contract_id" = ("backend_contractmetric"."contract_id")) LIMIT 1), 0.0), 19, 20, "backend_contractmetric"."collection_name", "backend_contractmetric"."total_owners", "backend_contractmetric"."total_supply", "backend_contractmetric"."mint_date", "backend_contractmetric"."unique_holder_percentage", "backend_contractmetric"."creator_royalty_percentage", "backend_contractmetric"."total_volume_traded", "backend_contractmetric"."total_earned", "backend_contractmetric"."block_value", "backend_contractmetric"."market_cap", "backend_contractmetric"."fully_diluted_valuation", "backend_contractmetric"."circulating_supply", "backend_contractmetric"."exchanges"

total_volume_traded_change is a field on ContractMetric

I could remove the subqueries to see if that fixes it, but i think i already tried that.

Thanks

I removed the subqueries and but still get the same results.

do you want it??

ContractMetric.objects.filter(capture_date__range=[start_date, end_date]).count()

Thanks for the suggestion, but its not what im trying to do.

My goal is to group the records and then annotate with the sums, but for some reason it wont group :frowning:

what groups and what sums are you talking about?
Instead of just saying what you want without any explanation, give a proper and accurate explanation.

Ah sorry if my original post wasn’t clear.

I’m trying to group items on my ContactMetric model by contract_id or any field really that would work so that i can annoate over each of the items in the group.

when i use contract_sums = metrics.values('contract__id').annotate( the items are still not grouped, but are instead return as single items.

For other models i have used the same process using .values(<field_name>).annotate(... and this has worked, but for some reason it wont on this model.

    end_date = datetime.now().date() - timedelta(days=1)
    start_date = end_date - timedelta(days=29)
    
    metrics = ContractMetric.objects.filter(capture_date__range=[start_date, end_date])

    # Subqueries for latest and previous values
    latest_floor_price_subquery = metrics.filter(contract=OuterRef('contract')).order_by('-capture_date').values('floor_price')[:1]
    previous_floor_price_change_subquery = metrics.filter(contract=OuterRef('contract'), capture_date=start_date).values('floor_price')[:1]
    previous_unique_holders_subquery = metrics.filter(contract=OuterRef('contract'), capture_date=start_date).values('total_owners')[:1]

    contract_sums = metrics.values('contract__id').annotate(
        capture_count=Count('id'),
        total_volume_traded_change_sum=Sum('total_volume_traded_change'),
        latest_floor_price=Coalesce(Subquery(latest_floor_price_subquery, output_field=FloatField()), Value(0, output_field=FloatField())),
        previous_floor_price=Coalesce(Subquery(previous_floor_price_change_subquery, output_field=FloatField()), Value(0, output_field=FloatField())),
        floor_price_change_over_7_days=ExpressionWrapper(
            F('latest_floor_price') - F('previous_floor_price'),
            output_field=FloatField()
        ),
...

I’ve figured this out. There were some data across the records that didn’t match the previous date record, which was causing the grouping to fail.

I dont really understand why this was causing it, but it explictly got the lastest values for these fields as a subquery and that seems to solve my problem.

Thanks for the replies.

Tom.

use aggregate.

end_date = datetime.now().date() - timedelta(days=1)
start_date = end_date - timedelta(days=29)
    
metrics = ContractMetric.objects.filter(capture_date__range=[start_date, end_date]).aggregate(
 count=Count('*'),
 sum=Sum('total_volume_traded_change'),
 latest=Coalesce('floor_price', Value(0)),
 previous=Coalesce('total_owners', Value(0)),
)
for i in metics:
  i['change'] = i['lalatest'] - i['latest']
  print(i)