We are developing a system that serves as a platform to work with and configure any number of other client-specific projects. This has worked so far, but now that we implement more complex features this has started to turn into a mess that becomes more difficult to maintain and develop for over time. I am hoping someone can suggest an architecture/structure that will work better and is more pythonic/djangonic in nature. Unfortunately I cannot give a complete minimal working example as boiling down the code and removing client information itself would take several hours to days. If absolutely necessary I might do that, but cannot justify it at the moment.
We chose Django to handle the authentication system, session handling, MVC-like development, etc. In addition we included a model for projects
class Project(models.Model):
name = models.CharField("Same as database name", max_length=50)
that serves to inform Django about what other projects exist. Furthermore we use a custom user model
class CustomUser(AbstractUser):
allowed_projects = models.ManyToManyField(Project)
is_allowed_all = models.BooleanField(default=False)
to assign users to projects. This works orthogonal to Django’s permission system (since we have to check these permissions with custom code wherever needed) and is a possible point of improvement, but at least this is working and requires no code change for new projects.
What also works okay is having views that change depending on the project that is selected via a URL parameter, then fetching the specific content:
urlpatterns = [
path('<int:project_id>/', views.detail, name='detail'),
]
def detail(request, project_id):
context = get_project_specific_context(project_id)
return render(request, "projects/detail.html", context)
Of note here is that these views might fetch data from databases that Django doesn’t know about, or at least is not allowed to manage. For this we do not use any model classes and instead fetch data with simple hardcoded SQL queries on the project’s database.
As a result of that feature we do have the problem of populating test databases for the projects that are being created by fixtures in the Django test database during tests. We solved that with rerouting the database connections to hardcoded SQLite databases and manually populating them with data. Since these views only read data this was a sufficient solution.
Now the current biggest roadblock is implementing an API for CRUD operations on the models saved in the project databases. All project databases can be expected to be of the same structure (there is a seperate system managing them), but this structure is supposed to be completely independent of anything the Django project does. This is one problem, in that any changes to the structure has to be reflected in the models used by the Django project, but changes are rare and can be easily integrated. To ensure the API doesn’t produce inconsistencies during the necessary seperate deployment of changes, we can check on establishing the connection that the database’s version is the same as the version in the model classes.
To implement the API we use the Django REST Framework, which seems to do exactly what we need. We have models
class DataPoint(models.Model):
name = models.CharField(unique=True, max_length=255)
unit = models.CharField(max_length=20, blank=True, null=True)
class Meta:
managed = False
db_table = 'data_point'
and serializers
class DataPointSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = DataPoint
fields = ['name', 'unit']
ordering = ['id']
and viewsets
class DataPointViewSet(viewsets.ModelViewSet):
project = "project_1"
queryset = DataPoint.objects.using("project_1").all().order_by("id")
serializer_class = DataPointSerializer
and registering routes for each viewset
IMPLEMENTED PROJECTS = {
"project_1": {"data_point": DataPointViewSet}
}
routes = []
router = routers.DefaultRouter()
for prj_name, prj_routes in IMPLEMENTED_PROJECTS.items():
for route, viewset in prj_routes.items():
router.register(prj_name + '/' + route, viewset)
routes += router.urls
return routes
The biggest problem with this is that the viewsets are project specific and hardcoded at that. That means a lot of copy&paste code if we want to enable the API for all projects. If we remove the using from the viewset, the API will try to fetch from the default database. Using a database router doesn’t work since it can only classify the read/write database on a model-class-level and has no information which viewset tries to fetch data in order so select the right database.
This problem with using a database router also strikes when trying to populate the test databases. In that regard we have no working solution yet and would try to get the API working before that anyway.
It is possible that we get something working with enough head-wall-banging, but is has become clear that the problems will only multiply with more features requiring the project databases. I hope someone might have an idea what we could improve, especially in regard to implementing the API on the project models.