I am experiencing a problem that might be related. For several days, I’ve been trying to establish a websocket connection using Django-channels on a Beanstalk environment in AWS and can’t find a solution. Could we continue discussing this on this thread? THANK YOU SO MUCH!!!
I am attempting to create a websocket through a web application in an AWS environment (Beanstalk). The components involved are: Django Channels, Daphne, and Redis (Redis ElastiCache). We are using POSTMAN as a client (although we also use Python scripts), and this is our URL: wss://api.???.com/ws/tunnel/group_test/.
The client can connect to the socket without any issues and receives the welcome message. This is the consumer handler responsible for it:
class ChatConsumer(AsyncWebsocketConsumer):
async def connect(self):
try:
self.room_name = self.scope['url_route']['kwargs']['comm_name']
self.room_group_name = f'chat_{self.room_name}'
# Unirse a un grupo de chat
await self.channel_layer.group_add(
self.room_group_name,
self.channel_name
)
await self.accept()
welcome_message_1 = f'Bienvenido al WebSocket 1. Tu canal es {self.channel_name}'
await self.chat_message({'message': welcome_message_1})
As we mentioned, the client successfully connects, but when sending a message to an existing group (otherwise, it does not produce an error, which in itself is a clue), we receive the following error message from Daphne:
2024-01-16 03:46:40,053 No-IP daphne.server ERROR Exception inside application: unpack(b) received extra data.
Traceback (most recent call last):
(full log below)
This is the consumer handler:
async def receive(self, text_data):
try:
text_data_json = json.loads(text_data)
message = text_data_json['message']
# Enviar mensaje al grupo de chat
await self.channel_layer.group_send(
self.room_group_name,
{
"type": "chat_message",
"message": message
}
)
The log error occurs during the await.
Initially, I thought there was a problem with the data I was sending to the operation, and that’s why unpack(b) was not performing its job well. But now I understand that while this could be a first error, the problem might come upstream, perhaps in the way Redis and Django communicate. I haven’t been able to find information in the Django release notes about this.
We work with AWS Redis ElastiCache, which should be well configured, as the connection takes place and the client correctly receives the welcome message. (in addition to the typical write/read variable checks). However, it is possible that in a more complex operation (message broadcasting), they don’t communicate well with Daphne…or Django-channels, and this is where I get a bit lost as I don’t know the exact way they communicate with each other. The LOG does not provide further information.
QUESTION:
Has anyone encountered this problem?
Could someone confirm the compatibility between Redis 7.1 and Django-channels?
Attempts at Resolution:
PRIOR ISSUE
A prior issue that may be a clue is that in the configuration of CHANNELS_LAYERS, I used as backend:
‘BACKEND’: ‘channels_redis.core.RedisChannelLayer’
But for some reason, this caused immediate disconnection after connection (the client connected and disconnected after a second) due to an error in the lua scripts:
2024-01-13 15:48:13,640 No-IP daphne.server ERROR Exception inside application: Lua scripts without any input keys are not supported.
However, when changing the backend to pubsub.RedisPubSubChannelLayer, this problem disappeared and the connection is now persistent.
TESTS PERFORMED
All the mentioned tests WORK CORRECTLY in a local test environment, with the same requirements.txt as the AWS environment.
We hardcoded the message sent. Same error reported in the log.
We created a Python command to publish a message to the group (in this case, the client is the instance itself, so we rule out permission issues). Same error reported in the log.
We created a REST view to receive the message sending request and through this, we invoked the handler for a specific channel. IMPORTANT: With this, we rule out a problem in the group send handler, as we use the same handler that the connection process uses (chat_message), which works correctly in AWS. Same error reported in the log.
Context:
Framework and Versions:
This is my requirements.txt:
amqp==5.2.0
APScheduler==3.10.4
asgiref==3.7.2
attrs==23.2.0
autobahn==23.6.2
Automat==22.10.0
billiard==4.2.0
certifi==2023.7.22
cffi==1.16.0
channels==4.0.0
channels-redis==4.1.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
colorama==0.4.6
constantly==23.10.4
cryptography==41.0.7
daphne==4.0.0
Django==4.2.6
django-cors-headers==4.3.0
django-redis==5.4.0
djangorestframework==3.14.0
djangorestframework-simplejwt==5.3.0
drf-yasg==1.21.7
flower==2.0.1
gunicorn==21.2.0
humanize==4.9.0
hyperlink==21.0.0
idna==3.4
incremental==22.10.0
inflection==0.5.1
kombu==5.3.4
msgpack==1.0.7
packaging==23.2
Pillow==10.1.0
prometheus-client==0.19.0
prompt-toolkit==3.0.43
psycopg2-binary==2.9.9
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
PyJWT==2.8.0
pyOpenSSL==23.3.0
python-dateutil==2.8.2
python-dotenv==1.0.0
pytz==2023.3.post1
PyYAML==6.0.1
redis==5.0.1
requests==2.31.0
service-identity==23.1.0
six==1.16.0
sqlparse==0.4.4
supervisor==4.2.5
tornado==6.4
Twisted==23.10.0
txaio==23.1.1
typing_extensions==4.8.0
tzdata==2023.3
tzlocal==5.2
u-msgpack-python==2.8.0
uritemplate==4.1.1
urllib3==2.1.0
vine==5.1.0
wcwidth==0.2.13
zope.interface==6.1
Instaleed apps
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'drf_yasg',
'corsheaders',
'usuarios',
'channels',
'channels_redis',
]
Configuración en settings.py de Channel layers
CHANNEL_LAYERS = {
'default': {
'BACKEND': 'channels_redis.pubsub.RedisPubSubChannelLayer',
'CONFIG': {
"hosts": [os.environ.get('REDIS_CHANNEL_LAYER_URL')],
},
},
}
REDIS_CHANNEL_LAYER_URL is defined in .env: rediss://cacheback-mxxx54.serverless.euw3.cache.amazonaws.com:6379
ENVIRONMENT DETAILS (staging-LQM1lest)
We have a load balancer to redirect traffic to a WSGI server (port 8000 where everything works as expected) and the wss traffic to an ASGI server (Daphne) for asynchronous management. The security groups are configured correctly and allow traffic as expected. The Elastic Redis cache service has been enabled and is working correctly (we persist and retrieve information without any problem).
Initially, as the Backend for CHANNEL_LAYERS, channels_redis.core.RedisChannelLayer was selected, but the connection was not persistent (the client connected and disconnected after a second), which was solved by selecting channels_redis.pubsub.RedisPubSubChannelLayer.
Problema Específico:
El cliente (postman) es capaz de conectarse correctamente al canal (conecta y recibe mensaje de bienvenida del servidor) pero al enviar un mensaje surge el error al procesar eventos receive en el consumidor WebSocket.
LOG
El traceback completo es el siguiente:
2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG WebSocket ['172.31.9.79', 52020] open and established
2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG WebSocket ['172.31.9.79', 52020] accepted by application
2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG Sent WebSocket packet to client for ['172.31.9.79', 52020]
2024-01-16 03:46:33,677 No-IP daphne.ws_protocol DEBUG Sent WebSocket packet to client for ['172.31.9.79', 52020]
2024-01-16 03:46:39,226 No-IP daphne.ws_protocol DEBUG WebSocket incoming frame on ['172.31.9.79', 52020]
2024-01-16 03:46:40,053 No-IP daphne.server ERROR Exception inside application: unpack(b) received extra data.
Traceback (most recent call last):
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/routing.py", line 62, in __call__
return await application(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/sessions.py", line 47, in __call__
return await self.inner(dict(scope, cookies=cookies), receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/sessions.py", line 263, in __call__
return await self.inner(wrapper.scope, receive, wrapper.send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/auth.py", line 185, in __call__
return await super().__call__(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/middleware.py", line 24, in __call__
return await self.inner(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/routing.py", line 116, in __call__
return await application(
^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/consumer.py", line 94, in app
return await consumer(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/consumer.py", line 58, in __call__
await await_many_dispatch(
File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/utils.py", line 57, in await_many_dispatch
await task
msgpack.exceptions.ExtraData: unpack(b) received extra data.
2024-01-16 03:46:40,289 No-IP daphne.ws_protocol DEBUG WebSocket closed for ['172.31.9.79', 52020]