Relationships with foreign resources from a REST API?

Using Django
1

Hi! First time posting here :smiley:

English is not my first language so if anything isn’t properly exaplained I am happy to clarify. Also, I am a complete beginner with django and web dev in general.

I am trying to build an app to manage hierarchies in a organization. Each of these hierarchies is esentially a tree with a bunch of nodes, lets say there is a CEO, below there is a production manager (PM) and a marketing manager, below PM there are other role, etc. Now, each of these roles should be associated with a specific employee.

So for now I have something like this.

class Hierarchy(Model):
    name = CharField()

class Role(Model):
    name = CharField()
    parent_role = ForeignKey("Role", null=True)
    hierarchy = ForeignKey("Hierarchy")
    employee = ForeignKey("Employee")

class Employee(Model):
    name = CharField()
    age = IntegerField()

My problem is that I need to retrieve the information about the employees from a REST API, and create the relationships using the ids provided by the API. For the sake of the example, let’s assume this API allows me to filter employees by age range

I would like to create a model Employee that is actually not stored in my db, but serves as an interface to the API that I need to use, so that I can make the connection as transparent as possible in the rest of my code. If I want to filter all the Roles that have associated an Employee in a specific age range I would like to do that jus like I would t if the Employee model was stored in my local db.

I don’t know if this is a good idea or if this is even possible.

I have looked at https://docs.djangoproject.com/en/3.0/ref/models/options/#managed, I have been reading about custom managers, and thought about using in-memory dbs, but none of my ideas seem super clean.

So is there any transparent and clean way I could implement this in django?
Thanks in advance :slight_smile:

2

To answer your question first, yes - you could implement it in a couple of different ways.

If I were implementing it, my Employee model would contain whatever information I need to retrieve the information I want from that foreign API. (It could be some ID code, a fully-qualified name, whatever - it doesn’t matter.)

I’d then define a method in the Employee model that retrieves the information from that API. If I store the results of that API in an appropriate Python object, I would then be able to access those attributes in my templates.

Filtering the data (say by age range), where that age is only accessible through the api is going to be a very “expensive” (time consuming) operation unless there’s a way to do something like a query in that foreign api to filter by age range - if you can do that, then you could create a custom manager to perform your filtering using that query operation and use those results in your query as a filter for what you retrieve from the Employee model.

(Side note: You having your employee FK in the Role model seems backwards to me. Is your organization set up such that an individual can fill multiple roles? (i.e. Have two instances of Role, each pointing to the same Employee.) Usually, when you see this type of structure used, the Employee has an FK to a Role. This allows multiple people to fill a role - for example, a Production Manager might have 5 staff members under them in the hierarchy. In your model, you would need to have 5 instances of Role for that staff position, one for each employee reporting up to that PM. If you turn that around, and have the FK in the Employee model, then you can have 5 employees all referencing the same Role that reports up to that PM.)

Ken

3

Hi Ken, thanks for taking the time to answer

To work with a more concrete example let’s assume an API that acepts GET /employees/ to retrieve all the employes and GET /employees/{pk} would get me the employee with that specific pk in that db. So following your proposal I would just store this pk in my Employee model.

I would love to be able to do the following to get an employee with a specific name:

Employee.objects.get(name="John Doe")

But as I understand, to do that I would need to do something like this:

for employee in Employee.objects.all():
    # this would fetch the api using the stored pk
    employee.fetch_api()
    # this would be the python object that stores the data
    if employee.data.name == "John Doe":
        return employee

I understand that if the API doesn’t allow to filter employees by name I would need to fetch all the employees to perform the filter anyways, but I would like to use the Employee model just like I would use any other model, and implement all the logic regarding the API comunication somewhere else (maybe using a custom manager as you mention?).

I was also looking at db.backends to kinda “hide” the API calls to the rest of my app, but that just doesn’t work and besides it is completely beyond my skill level.

Regarding your side note, you might be correct :sweat_smile: My use case is a bit more convoluted and I tried to simplify it to provide a small context, but the situation you mention makes a lot of sense. I will have to look at it more closely.

4

Re: Using the Employee model “just like I would use any other model”.

<Opinion> Extremely unlikely in the general case.</Opinion>

Keep in mind that the ORM filter method and attribute-access notation (e.g. role__employee to access the Employee object given a role) work by building an SQL statement that is passed to the database to be processed. The work of actually filtering data or joining across tables is done by the database and not by the Python code. So in that sense, you don’t have control at that point.

Now, if you were to write a custom PostgreSQL extension (assuming your database is PostgreSQL) that performs the access to the foreign API and processes the data within the database, I could see smooshing something together that generally works. But I wouldn’t want to go in that direction.

However, once you sidestep the issue of related field accesses from other tables, you could implement a much more limited version of this using a custom manager. If you’re willing to limit access to the Employee model to only referencing it through direct methods on the Employee.objects class (something like Employee.objects.get(...) and Employee.objects.filter(...)), then you could create your own manager that overrides get, filter, and any other methods you want to use by implementing that functionality within your versions of those methods. You’ll also need to handle the related object reference syntax and anything else that may apply.

<Opinion>I can understand the desire, but I’d seriously question whether or not I’d want to go that route. There’s something to be said for making it extremely explicit that accessing Employee related data is very different from all other data, especially if other people are working on this as well. When it comes time to diagnose problems associated with that API, I think you’ll find it very helpful to have it explicitly clear when and where that API is invoked.</Opinion>

5

I see, the limitations of my approach are a lot clearer now, I didn’t realize that much of the work is done by the database itself as you mention.

This really started as a thought experiment, I guess the fact that Employee data is quite different from my other models is not something that should be “hidden”.

Thank you very much for your answers