Allow Meta class to have extra Database specific options

Django Internals ORM
1

Hi, I’m developing a driver for CrateDB. GitHub - surister/cratedb-django: CrateDB backend for Django

CrateDB’s SQL dialect has extra functionalities, I’ll give you two examples:

  1. you can create a table clustered in X shards, e.g: CREATE TABLE mytable (f TEXT) CLUSTERED INTO 6 SHARDS
  2. You can issue a refresh table mytable after DML operations to have data available after the refresh, otherwise it’s 1s by default.

Customizing these options would be very important for any CrateDB user, hence doing something like:

class MyModel(models.Model):
    myfield = models.TextField()
    class Meta:
        auto_refresh = True # automatically call REFRESH TABLE after DML operations
        clustered_into = 10

is needed.

Now, we cannot do that, since django.db.models.options.Options is not injected, it’s just imported and used, we can of course, import it and change it but that would change the Options for every model, this is undesirable because:

  • Can clash with future Meta options added by Django
  • Can clash with other Databases backend that use the same technique
  • In situations where several backends are used (can easily be the case with CrateDB, mixing Postgres and CrateDB is desirable), it can cause confusion as the user might incorrectly use a model from CrateDB to a Postgres Model and not get an error (meta option would not apply)

Proposal:

In django.db.base.ModelBase, the Option class is used as new_class.add_to_class("_meta", Options(meta, app_label))

Ideally, ModelBase implements a new def get_optionclass()->Option where later we do new_class.add_to_class("_meta", get_option_class(meta, app_label)).

That would allow us to have custom ModelBase with custom Options, and have my models
MyModel(CrateModel) just inherit from my database specific model.

Now, I open this ticket to ask for feedback on my problem and proposed solution to see if it makes sense to persue this and try to implement this myself in Django.

Current solution

For the record, my current solution is to create a MetaCrate(ModelBase) where I take the the custom attrs, remove them previous to new call and then after the object is created, return them.

CRATE_META_OPTIONS = (
    ("auto_refresh", False),  # Automatically refresh a table on inserts.
)


class MetaCrate(ModelBase):
    def __new__(cls, name, bases, attrs, **kwargs):
        crate_attrs = {}

        # todo document

        try:
            meta = attrs['Meta']
            for crate_attr in CRATE_META_OPTIONS:
                attr_name = crate_attr[0]
                if attr_name in meta.__dict__:
                    crate_attrs[attr_name] = meta.__dict__[attr_name]
                    delattr(meta, attr_name)

        except KeyError:
            # Has no meta class
            pass

        o = super().__new__(cls, name, bases, attrs, **kwargs)

        # Return back the crate_attrs we took from meta to the already created object.
        for k, v in crate_attrs.items():
            setattr(o._meta, k, v)
        return o

And then later have my class CrateModel(models.Model, metaclass=MetaCrate): ...

1 Like
2

Just to make sure I understand you correctly, would this API work for you:

class MyModel(models.Model):
    myfield = models.TextField()
    class Meta(CrateDBOptions):
        auto_refresh = True # automatically call REFRESH TABLE after DML operations
        clustered_into = 10

Or are you looking for something different?

1 Like
3

Without diving too deep, I think so, something else to take into account is that I’m already adding database specific functions and behavior to CrateModel(models.Model) example:

class CrateModel(models.Model, metaclass=MetaCrate):
    """
    A base class for Django models with extra CrateDB specific functionality,

    Methods:
        refresh: Refreshes the given model (table)
    """

    @classmethod
    def refresh(cls):
        with connection.cursor() as cursor:
            cursor.execute(f"refresh table {cls._meta.db_table}")

    class Meta:
        abstract = True

so I need to have a custom CrateModel anyway, maybe it’s just cleaner to only have to modify the ModelBase

1 Like