Capturing live data between Django databases

When I pass data via query (Python Shell) to the database, I can display the data on the web without refreshing the page. However, when an Ethernet cable, an internal default switch of the VM, and multiple databases are involved, Django can’t capture the data and display the data on the web.

When I plug an Ethernet cable between devices, data flows into my app because I use the multiple databases feature of Django in the settings.py. But I don’t understand why Django cannot capture the data asynchronously and push it to the web. What’s the conventional way to do this? Sender device has PostgreSQL data whereas receiver Django has MySQL.

I use signals.py to capture it, where the data flows into my app. When I do a query, I can retrieve the data I want, but I can’t display it on the web.

How? Using what tools or web interface?

Having the ethernet cable alone doesn’t cause data to flow.

It can, but there’s either a conceptual or terminology issue here needing to be resolved.

Things don’t just “happen” by themselves. There’s some process involved with causing these things to occur.

Most importantly, it’s not the database (neither PostgreSQL nor MySQL) that initiates data transfers. There’s some other process involved here, and that is where we need more clarity.

Python Shell (python manage.py shell) provides API, right? We can manipulate db from there. When I pass data from there, I can change the data on the web without refreshing the page.

For second question, I use Django multiple databases feature, which means I added the second database(PostgreSQL) in the settings.py. That allows me to connect to external db. When I ping , I can get packets. So it is fine.

I have two apps involved in this process. One of them receives the data, let’s call it App1. When I do query via Python Shell, I can retrieve those timestamps or any data that models.py of App1 has. App2 is responsible for displaying the data.

App1/signals.py tries to handle whenever PostgreSQL produces data (there is a large codebase that sends those data continuosly if ethernet cable is plugged) whereas App1/tasks.py updates the relevant App2/models.py’s attributes with App1/models.py’s attributes. Two attributes should be updated with new data without refreshing the page.

The problem is capturing those data. Once I capture them, I can display them. WebSocket works.

Not precisely, no. The Python shell gives you interactive access to the Django ORM API, but it does not provide any APIs of its own.

This is absolutely correct, yes.

Is that a question or a statement? If a statement, how are you doing it now?
If it is a question, yes. You can create a system that causes the browser to update the data being displayed without a page refresh. (It involves some JavaScript in the browser, but it can be done.)

PostgreSQL does not “produce” data. PostgreSQL stores and retrieves data upon request by a separate process.

This would all happen within the server. A browser has no part in this.

PostgreSQL stores the Device 1’s data which is the sender. MySQL stores the Device 2’s data which is the receiver. Both of them has Django but the databases are different.

When I can change the data on the web without refreshing the page using Django ORM API in Device 2’s Django. But when multiple databases involve such as Device 1’s PostgreSQL which is the external db, Device 2’s MySQL in Django cannot capture the data asynchronously. But when I do query in Device 2’s MySQL db using shell via ORM API, I can see that data I need is there!.

I am investigating why I cannot capture the data I need (remember I can update the desired value by passing dummy data via query so my WebSocket works!) and update the attributes. Is this a connection problem? Database configuration? Because I am confident that once I capture data, I can display them.

Again, is this a statement or a question? If a statement, please expand on what you’re doing. If it’s a question, what is the question?

Again, is this a question or a statement? If it’s a question, yes, this can be done.

Almost certainly, it’s a code problem. If you’re connecting to both databases from the same Django project, then it’s neither a connection nor configuration issue.

But at this point we’re probably going to need to see the code involved to make any further progress.

Both of them are statement.

This is from App1’s models.py. These attributes are my target and contain live data from postgreSQL when ethernet cable is plugged.

class CommonTask(models.Model):
    command = models.IntegerField()
    date_started = models.DateTimeField(blank=True, null=True)
    date_done = models.DateTimeField(blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'common_task'

This is App1’s signals.py. The aim is to catch them (CommonTask’s attributes) whenever those attributes have new data.

from django.db.models.signals import post_save
from django.dispatch import receiver
from .models import CommonTask

from monorepo.tasks import commontask_postsave_task

@receiver(post_save, sender=CommonTask)
def common_task_post_save_handler(sender, instance, created, **kwargs):
	if created:
		commontask_postsave_task.delay(instance.command, instance.date_started, instance.date_done)

This is App1’s tasks.py. It supposed to update the current timestamps with caught attributes (date_started & date_done).

from celery import shared_task
from Main.models import SubProcess


@shared_task
def commontask_postsave_task(command, date_started, date_done):
	if command == "1":
		sp = SubProcess.objects.filter(name='Cut Ply').first()

		if sp:
			if date_started:
				sp.jobStart = date_started
			if date_done:
				sp.jobEnd = date_done

			sp.save()

This works when I pass data via ORM API. But it doesn’t work when PostgreSQL data involves.

This is App2’s signals.py. It supposed to send a WebSocket message about a related model instances to a specific group whenever that model istance is saved or updated.

from asgiref.sync import async_to_sync
from channels.layers import get_channel_layer

from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver
from .models import SubProcess

@receiver(post_save, sender=SubProcess)
def subprocess_handler(sender, instance, **kwargs):
	channel_layer = get_channel_layer()
	room_group = f"SubProcess_{instance.process.id}"

	message = {
		"Name": instance.name, 
		"fields": {}
	}

	dt_format = '%b. %d, %Y, %I:%M%p'

	if instance.jobStart:
		message['fields']['Job Start'] = instance.jobStart.strftime(dt_format)

	if instance.jobEnd:
		message['fields']['Job End'] = instance.jobEnd.strftime(dt_format)

	async_to_sync(channel_layer.group_send)(room_group, {'type':"update_message", "message":message})
		

Settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.mysql',
      'OPTIONS': {
          'read_default_file': '/etc/mysql/my.cnf',
      },
    },
    'xxx': {
        'ENGINE': 'django.contrib.gis.db.backends.postgresql',
        'NAME': 'xx',
        'USER': 'xxx',
        'PASSWORD': 'xx',
        'HOST': 'xx.xx.xx.xx',
        'PORT': 'xxxx',
        'POSTGRES_NAME':"xxx",
        'POSTGRES_USER':"xxx",
        'POSTGRES_PASSWORD' : "xx",
    }
}

Ok, there may be a misunderstanding here of what signals are in Django and how they work.

They do not trigger based upon any event. They are triggered within a specific instance of a Django process by code in that process performing the action that triggers the event.

Django signals are not a form of an IPC or a general event-monitoring system.

If you are looking to send updates out through a web socket, then the code that is saving the data to the database needs to forward the data through the channels layer in addition to writing the data to the database.

So, signals are redundant for this. How about Celery task in App1? Instead of signals.py, should I leverage Channels only? How should be the order or steps?

If both processes are sending data out through channels to go out via websockets, I’d send the data through the channels layer from each project (or celery task) as necessary.

If the desired flow of data is project 1 -> browser and project 2 -> browser, then I would have the appropriate code in each project sending the data through the channels layer at whatever point in the code that you want to do that.

I don’t understand project1 and project2 but I think you are suggesting that remove the signals and do the whole operation via Celery/tasks.py.

App1 gets the real-time data. App2’s signals.py sends data to Django consumer, and it transmits the WebSocket messages to frontend. I suppose you suggested that operations should be done by Celery/ tasks.py for each app instead of Django signals.

How? (What is it doing?) What is it doing with this data after it has gotten it?

How does Celery fit into this?

What does app2 do with this data? Is it intended to only serve as an intermediary to the consumers?

It may be better (and possibly easier) if you temporarily forget about your current implementation, and describe the architecture of the solution that you are trying to build. Don’t try to address this in terms of code, but describe what systems are involved, and what needs to happen among them.

Industrial PC has a PostgreSQL database. It has a Django project itself. It works with the PLC and the whole automation system. Whenever the process starts, IPC’s code produces timestamps at that moment.

My Django has MySQL. App1 and App2 belong to my Django. App1’s models.py are empty when the Ethernet cable is not plugged. I can see the IPC’s data in App1’s models.when the Ethernet cable is plugged. So clear, right?

App 2 is the main app. It contains the timestamps. App 1 has live timestamps from PostgreSQL. Those live timestamps need to replace App 2’s timestamps.

Signals had been thought of as incoming data handlers. But obviously, it will be removed. The thinking here is handling the incoming data in App1 and passing it to App2 so that we can display the real-time timestamps without refreshing the page (App2/templates) in real-time.

So it is basically a real-time data transfer from two different devices that have two different databases. I don’t understand the emphasis on general architecture. Data is in Django’s App1 already. The problem is how to convey it to the browser.

Celery’s issue is handling command attributes and timestamps periodically. It was thought of as the as the first stop for incoming data. Data needs to be captured first and then conveyed to App2.

The main issue is how to handle incoming data. What is the most conventional or solid way to do this?