[Rest API] How can Join two table and serialized the result

Hello,

I am coming to you because we spend all the afternoon to try to solve that issue.

I created an Rest API with Django Rest Framework.
I use the verb GET as the following http://127.0.0.1:8080/api/map/sensor/1/
My goal is to get all measure of the sensor with the ID 1.

my project/urls.py contain the following

from django.contrib import admin
from django.urls import path, include
#from map.viewsets import MarkerViewSet

urlpatterns = [
    path('', include('map.urls')),
    path('admin/', admin.site.urls),
    path('map/', include('map.urls')),
    path('api/map/', include('map.urlsapi')),
]

I created a urls file for the apis

from django.urls import path
from . import viewsapi
from map.viewsets import MarkerViewSet

app_name="map"
urlpatterns = [
    path("field/<int:idfield>/", MarkerViewSet.as_view({'get': 'list'}), name="marker_view_set"), # Work with Django Rest Framework
    path('sensor/<int:idsensor>/', viewsapi.sensor, name='apiSensor'), # Api classic a Django
]

I also created a view file for the api, named viewsapi.py and here the content

from django.http import JsonResponse
from rest_framework.parsers import JSONParser
from datetime import date, timedelta

from .models import Stations, Measures, Sensors
from .serializers import StationsSerializer,MeasuresSerializer
import pickle

def sensor(request, idsensor):
    if(request.method == 'GET'):
        # Get the last measure date/time for that sensor
        latest_measure = Measures.objects.filter(sensors_id_sensor=idsensor).order_by('-measure_created').first()
        # Get all measures from 3 days before the last measure
        sensor_measures = Measures.objects\
            .filter(sensors_id_sensor=idsensor, measure_created__range=[latest_measure.measure_created - timedelta(days=3), latest_measure.measure_created])\
            .order_by('measure_created')\
            .select_related('sensors_id_sensor')

        print(sensor_measures.query)

        # serialize the task data
        serializer = MeasuresSerializer(sensor_measures, many=True)
        #print(serializer.data)

        # return a Json response
        return JsonResponse(serializer.data, safe=False)
    elif(request.method == 'POST'):
        # parse the incoming information
        data = JSONParser().parse(request)
        # instanciate with the serializer
        serializer = SensorsSerializer(data=data)
        # check if the sent information is okay
        if(serializer.is_valid()):
            # if okay, save it on the database
            serializer.save()
            # provide a Json Response with the data that was saved
            return JsonResponse(serializer.data, status=201)
            # provide a Json Response with the necessary error information
        return JsonResponse(serializer.errors, status=400)

Finally, I have my serializer.py file and I added the following
Note, SensorsSerializer seems to be usefulness

from rest_framework_gis import serializers
from rest_framework import serializers as ser
from .models import Stations, Measures, Sensors

class SensorsSerializer(ser.ModelSerializer):
    class Meta:
        fields = ("id_sensor","sensor_name")
        model = Sensors


class MeasuresSerializer(ser.ModelSerializer):
    """
    sensor_name = ser.SlugRelatedField(
        many=False,
        read_only=True,
        slug_field='id_sensor'
    )
    """
   # that's a try, but I am not sure, it helps
    sensor_name=SensorsSerializer(
        many=True,
        read_only=True
    )

    class Meta:
        fields = ("sensor_name","id_measure","sensors_id_sensor", "value", "measure_created")
        model = Measures

Description of the problem

If I load the url http://127.0.0.1:8080/api/map/sensor/1/ I get the following result

That’s fine, but I need to know the name of the sensor. Then I should see an additional line sensor_name:"bud 1"

Now, if you return to viewsapi.py file, there is print print(sensor_measures.query) and it prints the following

SELECT `measures`.`id_measure`, `measures`.`sensors_id_sensor`, `measures`.`collections_id_collection`, `measures`.`value`, `measures`.`measure_created`, `sensors`.`id_sensor`, `sensors`.`stations_id_station`, `sensors`.`sensor_types_id_sensor_type`, `sensors`.`sensor_name`, `sensors`.`sensor_longname`, `sensors`.`sensor_description`, `sensors`.`sensor_active`, `sensors`.`sensor_created`, `sensors`.`chart_style_id_chart_style`, `sensors`.`chart_pointStyle_id_chart_pointStyle`, `sensors`.`chart_borderWidth`, `sensors`.`chart_fill`, `sensors`.`chart_showLine`, `sensors`.`chart_pointRadius`, `sensors`.`chart_pointHoverRadius`, `sensors`.`chart_backgroundColor_id_chart_backgroundColor`, `sensors`.`chart_borderColor_id_chart_borderColor` FROM `measures` INNER JOIN `sensors` ON (`measures`.`sensors_id_sensor` = `sensors`.`id_sensor`) WHERE (`measures`.`measure_created` BETWEEN 2021-04-27 12:57:05 AND 2021-04-30 12:57:05 AND `measures`.`sensors_id_sensor` = 1) ORDER BY `measures`.`measure_created` ASC
[30/Aug/2022 20:02:13] "GET /api/map/sensor/1/ HTTP/1.1" 200 32970

we can read that the jointure is done with

INNER JOIN `sensors` ON (`measures`.`sensors_id_sensor` = `sensors`.`id_sensor`)

and we can see the sensor name

`sensors`.`sensor_name`, `sensors`.`sensor_longname`

It look that all work fine.

Now, I returned to my serialzer file and I added sensor_name

class MeasuresSerializer(ser.ModelSerializer):
    """
    sensor_name = ser.SlugRelatedField(
        many=False,
        read_only=True,
        slug_field='id_sensor'
    )
    """

    sensor_name=SensorsSerializer(
        many=True,
        read_only=True
    )

    class Meta:
        fields = ("id_measure", "sensor_name", "sensors_id_sensor", "collections_id_collection", "value", "measure_created")
        model = Measures

but the sensor name is not printed. why?
I have the same result as you can see in the attached picture, above.

I also tried to add
sensors.sensor_name
but I got an error

Field name sensors.sensor_name is not valid for model Measures.

Then, I tried to add
sensors_id_sensor.sensor_name
with the same error

Then I suppose that sensor_name is best way

class MeasuresSerializer(ser.ModelSerializer):
    """
    sensor_name = ser.SlugRelatedField(
        many=False,
        read_only=True,
        slug_field='id_sensor'
    )
    """

    sensor_name=SensorsSerializer(
        many=True,
        read_only=True
    )

    class Meta:
        fields = ("id_measure", "sensor_name", "sensors_id_sensor", "collections_id_collection", "value", "measure_created")
        model = Measures

but why it does not print sensor_name, and then how can I do that? What did I miss?

Many thank for your help

Hey there!
Can you also share the Measures and Sensors models?

Hello,
Thanks for your reply!!

I just observed observed that if I comment the following

"""
sensor_name=SensorsSerializer(
        many=True,
        read_only=True
    )
"""

it does not work any more, then it seem to be suefull.

`Can you also share the Measures and Sensors models?
Yes, here is

class Measures(models.Model):
    id_measure = models.AutoField(primary_key=True)
    #sensors_id_sensor = models.ForeignKey('Sensors', models.DO_NOTHING, db_column='sensors_id_sensor')
    sensors_id_sensor = models.ForeignKey('Sensors', models.DO_NOTHING, related_name='sensor_n', db_column='sensors_id_sensor')
    collections_id_collection = models.ForeignKey(Collections, models.DO_NOTHING, db_column='collections_id_collection')
    value = models.DecimalField(max_digits=11, decimal_places=4)
    measure_created = models.DateTimeField()

    class Meta:
        managed = False
        db_table = 'measures'

class Sensors(models.Model):
    id_sensor = models.AutoField(primary_key=True)
    stations_id_station = models.ForeignKey('Stations', models.DO_NOTHING, db_column='stations_id_station')
    sensor_types_id_sensor_type = models.ForeignKey(SensorTypes, models.DO_NOTHING, db_column='sensor_types_id_sensor_type')
    sensor_name = models.CharField(max_length=20)
    sensor_longname = models.CharField(max_length=45, blank=True, null=True)
    sensor_description = models.TextField(blank=True, null=True)
    sensor_active = models.IntegerField(blank=True, null=True)
    sensor_created = models.DateTimeField()
    chart_style_id_chart_style = models.ForeignKey(ChartStyle, models.DO_NOTHING, db_column='chart_style_id_chart_style')
    chart_pointstyle_id_chart_pointstyle = models.ForeignKey(ChartPointstyle, models.DO_NOTHING, db_column='chart_pointStyle_id_chart_pointStyle')  # Field name made lowercase.
    chart_borderwidth = models.IntegerField(db_column='chart_borderWidth', blank=True, null=True)  # Field name made lowercase.
    chart_fill = models.IntegerField(blank=True, null=True)
    chart_showline = models.IntegerField(db_column='chart_showLine', blank=True, null=True)  # Field name made lowercase.
    chart_pointradius = models.IntegerField(db_column='chart_pointRadius', blank=True, null=True)  # Field name made lowercase.
    chart_pointhoverradius = models.IntegerField(db_column='chart_pointHoverRadius', blank=True, null=True)  # Field name made lowercase.
    chart_backgroundcolor_id_chart_backgroundcolor = models.ForeignKey(ChartBackgroundcolor, models.DO_NOTHING, db_column='chart_backgroundColor_id_chart_backgroundColor')  # Field name made lowercase.
    chart_bordercolor_id_chart_bordercolor = models.ForeignKey(ChartBordercolor, models.DO_NOTHING, db_column='chart_borderColor_id_chart_borderColor')  # Field name made lowercase.

    class Meta:
        managed = False
        db_table = 'sensors'

Here is the full models.py file

Try like this:

class MeasuresSerializer(ser.ModelSerializer):

    sensors_id_sensor=SensorsSerializer(
        read_only=True
    )

    class Meta:
        fields = ("sensors_id_sensor", "id_measure","sensors_id_sensor", "value", "measure_created")
        model = Measures

If it works, it may produce a nested json

Houhaa, thank, it’s look really better!!!

but why did you add twice sensors_id_sensor?

I tried the following

class MeasuresSerializer(ser.ModelSerializer):

    sensors_id_sensor=SensorsSerializer(
        read_only=True
    )

    class Meta:
        fields = ("id_measure", "sensors_id_sensor", "collections_id_collection", "value", "measure_created")
        model = Measures

and here the result

Great! but would have a way to have only the sensor name?
instead of

id_measure:492901
sensors_id_sensor
  id_sensor:1
  sensor_name:"b1"
value:"21.770"
measure_created:"2021-04-27T13:16:52Z"
id_measure:492901
sensor_name:"b1"
value:"21.770"
measure_created:"2021-04-27T13:16:52Z"

I tried

 fields = ("id_measure", "sensors_id_sensor.sensor_name", "collections_id_collection", "value", "measure_created")
 fields = ("id_measure", "sensors_id_sensor__sensor_name", "collections_id_collection", "value", "measure_created")

without success

The actual result is fine for me. So no worries if it’s too much
Many thanks

I heard, there were a way to create alias to have some name more easy to read, for example

id_measure:492901
sensor:"b1" #instead of sensor_name
value:"21.770"
created:"2021-04-27T13:16:52Z" #instead of measure_create

is right?

Great, so that works!
Now we can go into the tiny details!

Because that’s the field on your Measures that is a ForeignKey to the Sensors model.
This produced this output because we said that this field is a Serializer, DRF will serialize that into an Json Object.
And when we added it on the body of the serializer, we overrided the default field that DRF injects automatically. But we also need to add that field to the fields on the Meta.fields

To get the expected behavior that you want, you can do with a method field.
Try it out!

Take a look into source.

Dear @leandrodesouzadev
So many thanks for your help!!!
I am going to look for the links you provide me.

May I abuse and ask you a hint?
It’s fine to get all measures, but I need to filter by date range.
For example, I need all measures from 2021-04-27T13:16:52Z and 2021-05-27T13:16:52Z.

My colleague told me I could use a filter. Here is my note GET server/api/v1/field/1/sonsor?filter=type:2

my idea is

27.0.0.1:8080/api/map/sensor/1/?filter=start:"2022-05-01 10:00:00",end:"2022-06-30 10:00:00"

Is a good way to start? and how to get the date in my viewsapi.py file?

Thanks for the extra

No worries!
For filtering, check the documentation.

I don’t think that this is a valid query parameter:

But you can get away with something like this:
?filter=2022-05-01 10:00:00, 2022-06-30 10:00:00
Then, on the view you’ll need to split this string on the comma to get the start-end.

[opinion]
I think it’s better to set two different query parameters:
?start=2022-05-01 10:00:00&end=2022-06-30 10:00:00
[/opinion]

Dear @leandrodesouzadev

So many thanks, it look to works fine, excepted for a small thing.

I looked at the link your provide me, and I decide to rewrite my code as the follwoing

class SensorViewSet(viewsets.ReadOnlyModelViewSet):
    serializer_class = MeasuresSerializer

    def get_queryset(self):
        #get the params
        ids = self.kwargs['idsensor']

        
        latest_measure = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
        
        # Get all measures from 3 days before the last measure
        sensor_measures = Measures.objects \
            .filter(sensors_id_sensor=ids,
                    measure_created__range=[end.measure_created - timedelta(days=3),
                                            end.measure_created]) \
            .order_by('measure_created') \
            .select_related('sensors_id_sensor')

        print(sensor_measures)

        return sensor_measures

It works as before. Then I add the parameter section.

# http://127.0.0.1:8080/api/map/sensor/2/?start=2022-05-26%2010:10:00&end=2022-06-01%2010:10:00

class SensorViewSet(viewsets.ReadOnlyModelViewSet):
    serializer_class = MeasuresSerializer

    def get_queryset(self):
        #get the params
        ids = self.kwargs['idsensor']

        start = self.request.query_params.get('start')
        print("Start:")
        print(start)

        end = self.request.query_params.get('end')
        print("End:")
        print(end)

        if end is None or len(end) <= 0:
            # Get the last measure date/time for that sensor
            end = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
            print("End2:")
            print(end)
        else:
            end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S')
            print("End3:")
            print(end)
        
        #if start is None:

        print("Start2:")
        print(start)

        #keep it for a short time
        #latest_measure = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
        
        # Get all measures from 3 days before the last measure
        sensor_measures = Measures.objects \
            .filter(sensors_id_sensor=ids,
                    measure_created__range=[end.measure_created - timedelta(days=3),
                                            end.measure_created]) \
            .order_by('measure_created') \
            .select_related('sensors_id_sensor')

        print(sensor_measures)

        return sensor_measures

It’s work, excepted for the date I get from the params
Initialy, I get the lasted recored measure with

end = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()

and I can print the value as the following print(end.measure_created)

The problem is when I get the measure from my params: end = self.request.query_params.get('end'). It’s a string!

Then my idea, was to convert to an object:
end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S')

but end are not the same

end = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
print(end.measure_created) #print the date
end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S')
print(end) #print the date

Is a way, and how, to work around this datetime.strptime(end, '%Y-%m-%d %H:%M:%S') to save the value in end.measure_created?

In that way, I can keep the following code as the following

sensor_measures = Measures.objects \
            .filter(sensors_id_sensor=ids,
                    measure_created__range=[end.measure_created - timedelta(days=3),
                                            end.measure_created]) \
            .order_by('measure_created') \
            .select_related('sensors_id_sensor')

Many THANKS ! :slight_smile:

The problem here is that end is becoming two different objects in this if statement. I think that what you want is to take the measure_created from the last register when there is no query parameter end, and not transform end in a Measures object. So in the end you will end up with a datetime object, either from the query parameter or from the last register.

        if end is None or len(end) <= 0:
            last_measure = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
            end = last_measure.measure_created
        else:
            end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S')

One question is, you’re not using the start query parameter. Is that correct?

Dear @leandrodesouzadev

Once again, thank for your help
I tried that

if end is None or len(end) <= 0:
            last_measure = Measures.objects.filter(sensors_id_sensor=ids).order_by('-measure_created').first()
            end = last_measure.measure_created
        else:
            end = datetime.strptime(end, '%Y-%m-%d %H:%M:%S')

But I do not remember why I did not keep it. The result was not shown, but let me try it again,
sorry

One question is, you’re not using the start query parameter. Is that correct?

Is partially correct :slight_smile: . I started with end, when working, I will work with start because it need more if statement :slight_smile:

Let me try again