User-defined data structures

Using Django Using the ORM
1

I have to make a decision… I’m making a kind of dashboard application, in which each row is a product and each column is how well it meets a certain set of criteria.

So imagine a column for testing, which could have status values (a level) according to the following criteria:

Level 1
has some automated tests
manual tests are performed on release
Level 2
is covered with extensive unit tests
Level 3
automated test coverage is complete
is well covered by functional tests

Bur over time, there will be more columns for e.g. performance, or documentation, or security.

Or we could add a Level 4, or retrospectively add criteria for Level 3.

And in the future, someone might choose to use another deployment of the same application to track completely different criteria.

Would you:

  1. hard-code everything in straightforward models and migrate when required, and worry about making it extensible at some future time
  2. allow sets of criteria to be defined and managed by an admin user, and associated with product rows (with some rather unwieldy M2M footwork)
  3. use a library that allows an admin user to define the sets of criteria by defining models via the admin (and which library)?
2

It would be really tough for someone not familiar with this project to give you valid advice. I’d want to know a lot more about how this data is being used, and in practical / realistic terms, how many separate deployments are envisioned and at what time scale.

My gut reaction is that either #1 or #2 are perfectly valid options under different sets of circumstances. (And with the proper structures, I don’t believe the data manipulations would be as difficult as you may think.)
(I would also say that I would never consider #3 in any circumstance. Django is not my tool of choice for user-driven table structures environments.)

3

Hi Daniele, nice to see you around :wave:

I would personally start with a set of hardcoded status level as doing it through a many-to-many of the form

class TestStatus(models.Model):
    level = models.SmallIntegerField()
    description = models.Text()

class Product(models.Model):
    test_statuses = models.ManyToMany(TestStatus, related_name="product")
    ...

Will make it quite hard to efficiently determine what’s the maximum level met (which I assume is of interest in your dashboard) through for each product (assuming you will list them) through SQL as this thread exemplifies.

In other words, it’s quite hard to craft a proper and efficient query to annotate the maximum level met for all existing test statuses.

You need something to generate SQL alike to

SELECT
    (
        SELECT status
        FROM test_status
        LEFT JOIN product_test_statuses ON (
            product_test_statuses.teststatus_id = test_status.id
            AND product_test_statuses.product_id = product.id 
        )
        GROUP BY status
        HAVING bool_and(product_test_statuses.id IS NOT NULL)
        ORDER BY status DESC
        LIMIT 1
     ) max_met_status
FROM product
4

Thanks @charettes @KenWhitesell. It’s clear just how many orders of magnitude more simple it is to hard-code the relationships of projects and their attributes.

The size of the data means that I am not particularly worried about needing to perform efficient queries.

While exploring this I am mostly discovering how rusty some of my Django is…

I ran into a ValueError when running a migration (you cannot alter to or from M2M fields, or add or remove through= on M2M fields). Is there a reason why this can’t be caught at makemigrations time, rather than when it’s too late?

5

If you’re referring to the ability to add a custom through to a pre-existing ManyToManyField it has been in the work for a while already but alas it never crossed the finish line.

As for migrating from any other field to many-to-many one I’m afraid it would be quite hard to to do the right thing all the time for the developer it could be represented in many different ways as distinct model.

You’re best bet is to create a new field, create a data migration to backfill it, and then drop the old one and rename the new one.