my first websocket

Using Django Async/Channels
1

I want to add a dynamic “status” (off, on, booting, booted, crashed?) indicator to an otherwise static html page. 1 second latency is a good target. I don’t have a clear plan on how this “status” will be tracked, but I’ll be thrilled to get something working and refine it later.

I have 10 raspberry pi behind an nginx/django server.
(10x pi) — switch — debian/nginx/django — internet

pictures and stuff:

django stuff

the access page will take you to live instances (which are generally up, unless I am mucking around)

each pi is powered by PoE from the switch. I use SNMP to turn them off and on again:
github .com/CarlFK/pici/blob/main/ansible/roles/site/files/pib/snmp_switch/src/snmp_switch/views.py#L16
(I can’t post more than 2 links, thus the broken links.)

or a CLI when I am doing stuff on the server:
github .com/CarlFK/pici/blob/main/ansible/roles/site/files/pib/snmp_switch/scripts/poe.sh
which I expect to re-code by adding argparse to
github .com/CarlFK/pici/blob/main/ansible/roles/site/files/pib/snmp_switch/src/snmp_switch/utils.py

There are plenty of places I can hook into the pi’s power off/on messages, the dhcp server and the pi’s startup scripts. MQTT is feeling like what I should use for this part.

In case it isn’t clear, I’m not sure what I should be doing, in addition to how to do it.

poking around it seems I want to use Django-Channels, spent a bit of time reading channels.readthedocs .io but it’s a bit daunting so I’m hoping someone can point me to something similar to what I should be doing. I half expect it will be one of the snippets from channels.readthedocs

2

I can’t think of any code that I can point you to, but if you search this site for any of:

  • Raspberry Pi channels
  • IOT channels
  • redis pi channels

you’ll find some threads here discussing some of the various possibilities.

This will give you an infrastructure for allowing your server (running Django) to update the browsers connecting to it, to show you the status of your devices.

How you manage the devices from your server is a different issue. You’ll need to decide how you really want to test your devices to determine their status. (You could configure a script on each Pi to run periodically to send its status, or you could use SNMP to pull the status from the device. It’s really up to you to decide what you actually need for this.)

If you really wanted to get fancy, you could install something like Icinga on your server and the icinga client on the Pi. However, running those tests at 1-second intervals could create a bit of a load on the Pi. (That’s something you would need to check.) (Actually, running 10 tests / second on Icinga isn’t something I’d generally recommend either.)

Basically, the sky’s the limit here. Once you identify what your actual requirements are, it’ll be easier to narrow this down to a target architecture.

3

The driving force behind this pi status thing is me waiting from power on to the pi to booted so I can ssh in.

getting pi status:
I’m pretty sure I will use MQTT coming from all 3 of:

  1. PoE off/on code
  2. dhcp server - dnsmasq that has a hook for events - example: pici/tools/dhcp/dhcp-logger.py at main · CarlFK/pici · GitHub
  3. on the pi, systemd - as soon as networking is up, after sshd is ready, when shutdown has started, and maybe right before networking goes down.

mqtt means there is an event loop to process:
mqtt.client.message_callback_add("pi/status", self.pi_status)
I have used:
mqtt.client.loop_forever()
except I bet I am going to need:
rc = mqttc.loop(timeout=1.0)

I kinda get that nginx has an event loop processing incoming http, which sometimes passes things to
proxy_pass http://unix:/run/gunicorn.sock;
gunicorn is another event loop (I think?) and then my Django view code gets called and needs to return something before things timeout.

I’m not sure I need Django for this - yet. I don’t need auth, I don’t need the ORM.

I’d like to say I should consider dev/testing. except the hardware requirements make it hard to do anything local, so the production equipment is where r&d and testing actually happen.

Any advice on starting with Django-Channels vs… something simpler?

4

I think first, you need to really nail down what it is you’re trying to do here. My take on what you’ve written is that some parts of this are still kinda hazy. (My apologies if I’m misreading your post.) I might suggest you define:

  • What devices are sending data. (You’ve identified a PoE switch, a dhcp server, and 10 Pi so far)
  • What data is being sent.
  • Who is receiving that data.
  • What is being done with the data after it has been received.

Once you can fully identify your requirements, then it’s worth defining an architecture for it. But trying to identify components without fully understanding what you’re trying to do can be extremely problematic.

As a side note, I’m having a difficult time understanding what benefit you’re going to achieve by using mqtt in this environment. Given that these devices are only being managed by (ior sending information to) one device, I don’t see a value in setting up a broker to handle it.

5

yeah, it is hazy here too, but I think I have enough to both make something useful and learn this new thing.

The PoE switch isn’t sending data, but the SNMP python code that controls the switch - I can add code to send status somewhere. I can also poll the switch for poe on/off, but I don’t think that is necessary.

All the status sources will send a pi ID (1,2,3…10) and a status (off, on, dhcp, ssh, shutdown) “on” means powered, but hasn’t booted enough to be useful, which takes from 30 seconds to 2 min depending on I don’t know what.

the server is receiving the status, the status is being pushed to the browser to update the page the user is looking at.

MQTT because I am not sure how else to get the pi status into the process that is pushing data to the browser. I’ve used it and am comfortable with callbacks which I’m guessing is needed to have the server push something to the client.

I can do an http get or post to an api endpoint, but from what little I have seen, that will be a typical django view that just returns a response to the client. I can put the status in to the orm/db, but I don’t see how it will get pushed to the browser.

Maybe another websocket thing can be used to send status, but that doesn’t seem like a good fit. (based on something I know very little about)

6

Ok, break this down into the separate and discrete steps.

Getting data from the device to your server - Multiple options are available.
In a controlled environment such as this, a udp packet is really all you need. Each device sends the data in whatever format you design to whatever port you wish to use. The listener on the server side listens to that port and forwards the data somewhere. No need for any heavyweight protocol or external or additional broker.

If you’re using Channels to support websockets to the browser, then your listener can forward the data through the channel layer to a Channels group. The Channels group mechanism would then forward that data to all connected consumers.

7

The listener on the server side listens to that port

I’m not sure how to do that without blocking. somehow I need to get the status data into the event loop (or at least memory space?) that pushes to the client.

I’m not sure how many hops there are here.

is “somewhere”
A) the browser via Channels
or b) the code that listens for data and then uses Channels to send to the browser?

Either way, I’m not sure how to do this part.

On the plus side, You have helped me figure out what I should be doing on the server/browser bit.

8

Nope. A blocking loop is fine in this case. This doesn’t need to be complicated or intricate.

It blocks on reading the port. When a packet comes in, it writes that packet with any reformatting or other information you wish to add to it to the Channel layer. Once that packet is written, it blocks on the port until the next packet comes in.

Something like this:

[device] -- udp --> [listener] -- channel --> [Channels consumer] -- websocket --> [Browsers]
9

I’m off to a good start: this works

browse 127.0.0.1:8000/static/pistat.html
type text, hit send, it shows up. yay and thank you for helping me get this far.

Now I am trying to figure out how to send a message from some python code.

https://channels.readthedocs.io/en/latest/topics/channel_layers.html#using-outside-of-consumers

but I am struggling to make something that works. not only does it not work, but it doesn’t work in different ways depending on if I run send.py or cut/paste the lines into the repl. So before I spend more time, am I even on the right track?

(pib) carl@x1:~/src/tv/pib/pici/ansible/roles/site/files/pib$ cat pistat/send.py 
# pistat/send.py
# sends a message to the pistat websocket thing

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pib.settings")

from channels.layers import get_channel_layer

channel_layer = get_channel_layer()
print( type(channel_layer) )

from asgiref.sync import async_to_sync
async_to_sync(channel_layer.group_send("pi2", {"type": "chat.system_message", "text": "hello"} ) )

(pib) carl@x1:~/src/tv/pib/pici/ansible/roles/site/files/pib$ python3 
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pib.settings")
'pib.settings'
>>> from channels.layers import get_channel_layer
>>> channel_layer = get_channel_layer()
>>> print( type(channel_layer) )
<class 'NoneType'>
>>> from asgiref.sync import async_to_sync
>>> async_to_sync(channel_layer.group_send("pi2", {"type": "chat.system_message", "text": "hello"} ) )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'group_send'
>>> 

(pib) carl@x1:~/src/tv/pib/pici/ansible/roles/site/files/pib$ python3 pistat/send.py 
Traceback (most recent call last):
  File "/home/carl/src/tv/pib/pici/ansible/roles/site/files/pib/pistat/send.py", line 9, in <module>
    channel_layer = get_channel_layer()
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/channels/layers.py", line 357, in get_channel_layer
    return channel_layers[alias]
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/channels/layers.py", line 78, in __getitem__
    self.backends[key] = self.make_backend(key)
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/channels/layers.py", line 43, in make_backend
    config = self.configs[name].get("CONFIG", {})
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/channels/layers.py", line 37, in configs
    return getattr(settings, "CHANNEL_LAYERS", {})
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/carl/.virtualenvs/pib/lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pib'
(pib) carl@x1:~/src/tv/pib/pici/ansible/roles/site/files/pib$
10

The async_to_sync call wraps the function, not what the function calls. The parameters need to be passed to what the async_to_sync function returns. In other words, it’s written like this:

async_to_sync(an_async_function)(parameters_for_async_function)

It may be easier to visualize what it’s doing by separating it into two lines:

wrapped_function = async_to_sync(an_async_function)
wrapped_function(parameters_for_async_function)

(Not that I’ve ever seen it actually used that way.)

11

progress!

I can run 2 instances of mange.py shell and send messages between them! woot!

>>> async_to_sync(channel_layer.send)('test_channel', {'type': 'hi'})

>>> async_to_sync(channel_layer.receive)('test_channel')
{'type': 'hi'}

except I can’t get this to work with the js code.

I’m guessing i need groups?

https://channels.readthedocs.io/en/latest/topics/channel_layers.html#groups

I tried, and now even my js code doesn’t work.
connect fires, but not .receive

um… help?

# pistat.html
 
       const logSocket = new WebSocket(
            'ws://'
            + window.location.host
            + '/ws/pistat/'
            + 'pi2'
            + '/'
        );

# pistat/routing.py
  websocket_urlpatterns = [
      re_path(r"ws/pistat/(?P<pi_name>\w+)/$", consumers.PiStatConsumer.as_asgi()),

# pistat/consumers.py
import json

from pprint import pprint

from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync

class PiStatConsumer(WebsocketConsumer):

    def connect(self):
        pprint(self.channel_name)
        self.channel_layer.group_add("pistat", self.channel_name)

    def disconnect(self, close_code):
        self.channel_layer.group_discard("pistat", self.channel_name)

    def receive(self, text_data):

        pprint(text_data)

        self.channel_layer.group_send(
            "pistat",
            {
                "type": "message",
                "text": text_data,
            },
        )

    def message(self, event):
        self.send(text_data=event["text"])

    def notify(self, event):
        print("notify:")
        print(event)
12

If you haven’t worked your way through the Channels tutorial, now’s a good time for that. The chat app shows how to work with groups in Consumers, and covers a lot of other fundamental topics along those lines.

One critical concept to always keep in mind: There’s a huge difference between channel_layer.send and a_consumer.send. The send method in a consumer sends data out the websocket to the browser. The send method in the channel_layer is what sends data to a consumer.

In the direct case:
ws = websocket
cl = channel_layer

[browser] -- ws --> [consumer] -- cl.send --> [consumer] -- consumer.send (ws) --> [browser]

Groups are what add the ability to send to multiple consumers, where the consumers are not known to the sender. The sender doesn’t need to know who else is in the group.
Also, unless the sender needs to receive a copy of the broadcast message, the sender does not need to be a member of the group.

The mechanics of implementing all this are in the tutorial.

13

great success!

I can put web socket on my cv :stuck_out_tongue:

Thank you for taking the time to help me. before I posted here I only had a hunch web socket was what I wanted (based on a few friends vague advice.) I came across Django-sockets which seemed like what I wanted except for some sort of Do Not Use message somewhere. Django-Channels is exactly what I need, but at first it seemed overly complex. Your posts here assured me this was what I should spend time on. So thanks again.

1 Like