UnicodeDecodeError triggered in serializing data of charset utf8mb4

Using Django Mystery Errors
1

We have a run-time error on the server, but the issue does not happen in the development environment.

The error rises from the below function calls in our code, and the database is in the character set utf8mb4, converted from the character set latin1 the last year.

Other servers running the same software are OK, and the error happens on this one only.

  File "/data/group/APPLICATION/applicationapp/rest/views.py", line 97, in get_reg_products
    obj = rest_models.FastTransactionSerializer.pre_init(businesses, product_id, offset_transaction_id)
  File "/data/group/APPLICATION/applicationapp/rest/models.py", line 189, in pre_init
    for value in value_qs:

Because the error does not happen anywhere else, we probably need to debug the live server step by step. So, we will also highly appreciate any tips on debugging a live remote Linux server. We have PyCharm Professional, but other tools are also OK with us.

It could also be someone who made a mistake on the server, but the log files were gone for the days. We don’t want to trace anybody to blame but we are under pressure to fix the issue.

Any hits or suggestions will be highly appreciated. Also, if you need more details, just let me know.

The exception trace in Gunicorn log:

==> ./applicationapp.log <==
[03/Feb/2022 13:27:09] ERROR [django.request:230] Internal Server Error: /api/products/reg/206/
Traceback (most recent call last):
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/data/group/venv3.7/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/data/group/venv3.7/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/data/group/venv3.7/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/data/group/venv3.7/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/data/group/venv3.7/lib/python3.7/site-packages/rest_framework/decorators.py", line 50, in handler
    return func(*args, **kwargs)
  File "/data/group/APPLICATION/applicationapp/rest/views.py", line 97, in get_reg_products
    obj = rest_models.FastTransactionSerializer.pre_init(businesses, product_id, offset_transaction_id)
  File "/data/group/APPLICATION/applicationapp/rest/models.py", line 189, in pre_init
    for value in value_qs:
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/models/query.py", line 287, in __iter__
    self._fetch_all()
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/group/venv3.7/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/data/group/venv3.7/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/data/group/venv3.7/lib/python3.7/site-packages/MySQLdb/cursors.py", line 321, in _query
    self._post_get_result()
  File "/data/group/venv3.7/lib/python3.7/site-packages/MySQLdb/cursors.py", line 355, in _post_get_result
    self._rows = self._fetch_row(0)
  File "/data/group/venv3.7/lib/python3.7/site-packages/MySQLdb/cursors.py", line 328, in _fetch_row
    return self._result.fetch_row(size, self._fetch_type)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 125: invalid start byte
2

That’s an indication of a data error within the table. You have a row with a character with a hex value of 0x96 - which is what Microsoft uses as the “en-dash”. (Likely cause is someone copy-pasted text from a MS Word or Excel document into a text field.)
Find the row and clean up the field.

1 Like