Upgrade of Python+Django triggered AttributeError: 'list' object has no attribute 'strip'

Using Django
1

I’m upgrading a system from Python2.7+Django1.11 to Python3.7.9+Django3.1.4.

This logic path worked OK on the older versions; but on the newer version it triggers an error saying AttributeError: 'list' object has no attribute 'strip'.

Update:

The direct trigger was Null value in result = parse_datetime(value.strip()), and culprit might be in my own source code due to behavior change after upgrade.

I’ve narrowed down to a single line in source code below:
After upgrade, option_2 = self.instance.product_template started returning wrong result. However, if set a break point in debug mode to briefly pause at the statement, the result will be correct again.

I suspect it’s related to system behavior of object initialization and concurrency; any pointers will be highly appreciated.

Part of my application source code:

class ProductAdminForm(BaseDynamicEntityForm):
    """
    This is an admin form which uses the EAV class
    """
    model = models.Product
    def _build_dynamic_fields(self):
        """
        this is an integration point, overrides existing method in third party source
        :return:
        """
        # reset form fields
        self.fields = deepcopy(self.base_fields)

        # Refactored for test:
        #product_template = self.initial.get('product_template') or self.instance.product_template
        #
        option_1 = self.initial.get('product_template')
        option_2 = self.instance.product_template
        product_template = option_1 or option_2

        # this line is the primary reason for override, additional filter on the fields by business

        # Refactored for test:
        #for attribute in self.entity.get_all_attributes().filter(producttemplateattribute__product_template=product_template):
        #
        all_attributes = self.entity.get_all_attributes()
        the_attributes = all_attributes.filter(producttemplateattribute__product_template=product_template)
        for attribute in the_attributes:
            value = getattr(self.entity, attribute.slug)
            ... ...

        return

Error message:

[18/Dec/2020 14:18:27] ERROR [django.request:230] Internal Server Error: /api/admin/APPLICATION/product/add/
Traceback (most recent call last):
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/contrib/admin/options.py", line 614, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/contrib/admin/options.py", line 1653, in add_view
    return self.changeform_view(request, None, form_url, extra_context)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/contrib/admin/options.py", line 1534, in changeform_view
    return self._changeform_view(request, object_id, form_url, extra_context)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/contrib/admin/options.py", line 1573, in _changeform_view
    form_validated = form.is_valid()
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/forms.py", line 177, in is_valid
    return self.is_bound and not self.errors
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/forms.py", line 172, in errors
    self.full_clean()
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/forms.py", line 374, in full_clean
    self._clean_fields()
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/forms.py", line 392, in _clean_fields
    value = field.clean(value)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/fields.py", line 149, in clean
    value = self.to_python(value)
  File "/data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/fields.py", line 471, in to_python
    result = parse_datetime(value.strip())
AttributeError: 'list' object has no attribute 'strip'
2

We’ll need to see the code. At a minimum we may need to see the form and view involved here. We may also end up needing to see the model.

When posting code, please enclose it between lines of three backticks - ```. This means you’ll have a line of only ```, then your code, then another line of ```.

Ken

3

Thank you for your reply, @KenWhitesell.

I’ve updated the post, and provided the relevant part of my application source code. Could you please take a look?

Thank you in advance for your help.

4

I’m having difficulty reconciling these two statements. If value is null, the error thrown is not going to throw an attribute error for type list.

Also, while you did post some parts of the view, I don’t see where it’s enough to follow your thoughts on where it’s happening, when and why. (Note: to the extent of my knowledge, there shouldn’t be any concurrency issues within the view unless your application is creating your own threads. Django view processing is effectively single threaded.)

Is this error being thrown when the form is being initially displayed, or when the data is being submitted? (The references to the validation leads me to believe the latter)

5

Thank you for your reply, @KenWhitesell.

I’ve tested again and got more information:

  1. Before triggering the exception, the debugging variable watching list shows value = {list: 2} [None, None]. Does this answer to your question?
The relevant source code in library file /data/APP-py3/venv3.7/lib/python3.7/site-packages/django/forms/fields.py:

    def to_python(self, value):
        """
        Validate that the input can be converted to a datetime. Return a
        Python datetime.datetime object.
        """
        if value in self.empty_values:
            return None
        if isinstance(value, datetime.datetime):
            return from_current_timezone(value)
        if isinstance(value, datetime.date):
            result = datetime.datetime(value.year, value.month, value.day)
            return from_current_timezone(result)
        try:
            result = parse_datetime(value.strip())
        except ValueError:
            raise ValidationError(self.error_messages['invalid'], code='invalid')
        if not result:
            result = super().to_python(value)
        return from_current_timezone(result)
  1. The exception happens when submitting a POST request.
Call stack in PyCharm:

to_python, fields.py:472
has_changed, fields.py:182
changed_data, forms.py:449
__get__, functional.py:48
construct_change_message, utils.py:503
construct_change_message, options.py:1055
_changeform_view, options.py:1582
changeform_view, options.py:1534
_wrapped_view, decorators.py:130
_wrapper, decorators.py:43
add_view, options.py:1653
inner, sites.py:233
_wrapped_view_func, cache.py:44
_wrapped_view, decorators.py:130
wrapper, options.py:614
_get_response, base.py:179
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
__call__, deprecation.py:114
inner, exception.py:47
get_response, base.py:128
__call__, wsgi.py:133
__call__, handlers.py:76
run, handlers.py:137
handle_one_request, basehttp.py:197
handle, basehttp.py:172
__init__, socketserver.py:720
finish_request, socketserver.py:360
process_request_thread, socketserver.py:650
run, threading.py:870
_bootstrap_inner, threading.py:926
_bootstrap, threading.py:890

Does the above information make sense? Please let me know if you need additional clarifications, and thank you for your help.

6

Yea, I think we would need to see the complete form. Both the form class itself and any base classes not part of the Django package.

And, if this is being handled by a user-written view, we’ll likely end up needing to see that, too.

(Primarily, what I’m thinking I need to see are the field definitions, any custom widgets that may be used, any clean_<fieldname> methods and/or the form clean method - and any user-written methods called by them.)

Unfortunately, there’s just not enough information from what you’ve posted so far to make any sort of diagnosis without going into the code in a lot more detail.

1 Like
7

Thank you for your reply, @KenWhitesell.

I’m attaching partial source code of class ProductAdminForm and class BaseDynamicEntityForm. Could you please see if it helps?

By the way, I suspect the call of super(BaseDynamicEntityForm, self).__init__(data, *args, **kwargs) has a synchronization issue: if I set a debug break point right behind it, the error will not happen; if removed the break point, the error happens. Please let me know if you want to test in this area.

If necessary, could you please give me an example of clean_ method?

Thank you for your patience and help, and for anything just let me know.

class ProductAdminForm(BaseDynamicEntityForm):
    """
    This is an admin form which uses the EAV class
    """
    model = models.Product
    def _build_dynamic_fields(self):
        """
        this is an integration point, overrides existing method in third party source
        :return:
        """
        # reset form fields
        print("admin _build_dynamic_fields: %s" % self.instance.product_template)
        self.fields = deepcopy(self.base_fields)

        #product_template = self.initial.get('product_template') or self.instance.product_template
        #
        option_1 = self.initial.get('product_template')
        option_2 = self.instance.product_template
        product_template = option_1 or option_2

        all_attributes = self.entity.get_all_attributes()
        the_attributes = all_attributes.filter(producttemplateattribute__product_template=product_template)
        # this line is the primary reason for override, additional filter on the fields by business
        #for attribute in self.entity.get_all_attributes().filter(producttemplateattribute__product_template=product_template):
        #
        for attribute in the_attributes:
            value = getattr(self.entity, attribute.slug)

            defaults = {
                'label': attribute.name.capitalize(),
                'required': attribute.required,
                'help_text': attribute.help_text,
                'validators': attribute.get_validators(),
            }

            datatype = attribute.datatype
            if datatype == attribute.TYPE_ENUM:
                enums = attribute.get_choices() \
                                 .values_list('id', 'value')

                choices = [('', '-----')] + list(enums)

                defaults.update({'choices': choices})
                if value:
                    defaults.update({'initial': value.pk})

            elif datatype == attribute.TYPE_DATE:
                defaults.update({'widget': AdminSplitDateTime})
            elif datatype == attribute.TYPE_OBJECT:
                continue

            MappedField = self.FIELD_CLASSES[datatype]
            self.fields[attribute.slug] = MappedField(**defaults)

            # fill initial data (if attribute was already defined)
            if value and not datatype == attribute.TYPE_ENUM: #enum done above
                self.initial[attribute.slug] = value

        return


class BaseDynamicEntityForm(ModelForm):
    '''
    ModelForm for entity with support for EAV attributes. Form fields are
    created on the fly depending on Schema defined for given entity instance.
    If no schema is defined (i.e. the entity instance has not been saved yet),
    only static fields are used. However, on form validation the schema will be
    retrieved and EAV fields dynamically added to the form, so when the
    validation is actually done, all EAV fields are present in it (unless
    Rubric is not defined).
    '''

    FIELD_CLASSES = {
        'text': CharField,
        'float': FloatField,
        'int': IntegerField,
        'date': DateTimeField,
        'bool': BooleanField,
        'enum': ChoiceField,
    }

    def __init__(self, data=None, *args, **kwargs):
        print("**kwargs: [[ ")
        for key, value in kwargs.items():
            print("%s == %s" % (key, value))
        print("]]")
        super(BaseDynamicEntityForm, self).__init__(data, *args, **kwargs)
        print("self.instance: %s" % self.instance)
        print("-------------------------")
        config_cls = self.instance._eav_config_cls
        self.entity = getattr(self.instance, config_cls.eav_attr)
        self._build_dynamic_fields()
8

See the docs for Cleaning a specific field attribute.

But what you’ve provided here is just a start. It does seem to indicate that there’s a lot of “dynamics” involved with this - which unfortunately also means that we’d need to see a lot more - including not just the complete forms involved but also the complete ModelAdmin classes and the complete models.

This is also the second time you’ve mentioned a possible synchronization issue. Django, by default, does not process views across multiple threads. If your code is doing any multiprocessing of its own, spawning off threads or processes, then sure - this is a possibility. But you’ve provided no code showing any indication of anything like this happening.

9

It looks like, the culprit was in ProductAdminForm._build_dynamic_fields(self). It had a customization adding widget AdminSplitDateTime to attributes in type of date. It splits DateTimeField into a list of strings, and in a later step triggered exception in _clean_fields(self).

            elif datatype == attribute.TYPE_DATE:
                defaults.update({'widget': AdminSplitDateTime})

Quote an online resource:

@alias51 The value returned by the AdminSplitDateTime widget is a list . The DateTimeField expects the value to be a string and calls .split() , which gives you this exception. – [Daniel Hepper]

This widget worked OK in earlier versions of Python and Django though.

Thank you for your pateince and help, @KenWhitesell, especially your reminder to check _clean functions. Happy holidays.

1 Like