Editing a field on a QuerySet

Using Django Using the ORM
1

I am processing documents with a prescribed title format that they begin with the year, month, and day in iso format. Due to a bug in my script, some of these documents got two dates instead of one. Many times, the dates are identical, but sometimes they are not. In the former case, I just want to cut off the first date. In the latter I want to select the older of the pair and remove the other.

Based on my understanding, this is not a use case for the update method, because while it is the same field, the correct values are not going to be the same, or, put another way, each instance needs a different correction.

I created a script that uses “__regex” to filter for entries with two dates at the beginning. That part works fine.

dd_title = eoa.filter(title__regex=r'\d{4}-\d{2}-\d{2}\s\d{4}-\d{2}-\d{2}')

The problems come after that line. In the first couple of runs, I got integrity errors, which I put in a try/except:

for t in dd_title:
    # these are full objects, not just the title attribute
    te = t.title
    if te[:10] == te[11:21]:
        a = te[11:]
        t.title = a
        try:
            Entry.save(t)
        except (psycopg2.errors.UniqueViolation, django.db.utils.IntegrityError):
            dd_title.exclude(t)
        continue

However, the integrity errors persisted:

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "ktab_entry_title_6950e951_uniq"
DETAIL:  Key (title)=(2023-06-06 ToT GDE insert) already exists.

and

django.db.utils.IntegrityError: duplicate key value violates unique constraint "ktab_entry_title_6950e951_uniq"
DETAIL:  Key (title)=(2023-06-06 ToT GDE insert) already exists.

So that was the first problem. I don’t understand why the try/except is not effective. I added the .exclude(t) and continue, thinking that would solve the problem, but it did not. I continued to get these same integrity errors. The difference now was that after reciting these same errors, the traceback added at the end:

TypeError: cannot unpack non-iterable Entry object

Entry is a model class. Its querysets are clearly iterable. So is this error speaking of a single member of the queryset?

Here are the two traceback lines immediately before the TypeError:

child_clause, needed_inner = self.build_filter(
                                ^^^^^^^^^^^^^^^^^^
 File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1268, in build_filter
   arg, value = filter_expr
   ^^^^^^^^^^
TypeError: cannot unpack non-iterable Entry object

I did some googling, but the **overwhelming number of serps **only speak of a NoneType error:

But this is not NoneType.

Now I have assigned different variable names to the different conditions my if clause might encounter. But they aren’t a sequence, they are conditionals. Is this nevertheless ‘iterating’ or ‘unpacking’?

"…When you’re working with iterable objects like Lists, Sets, and Tuples in Python, you might want to assign the items in these objects to individual variables. This is a process known as unpacking.

[Ok, I can see that…]

During the process of unpacking items in iterable objects, you may get an error that says: “TypeError: cannot unpack non-iterable NoneType object”.
This error mainly happens when you try to assign an object with a None type to a set of individual variables…"

freecodecamp, supra.

When and how did the string I am manipulating become a NoneType? Perhaps it did not. My actual error does not make that claim. It says it is an Entry object, but nevertheless not iterable.

The string we are working with is the title of the Entry object, i.e., Entry.title, or if you prefer, entry.title. It is an Entry object, but it has the value of ‘str’. I am iterating over the subset of Entry objects created by the use of the “__regex” filter, in other words, Entry objects whose title string is a match for my regular expression.

That can’t be the basis for the exception. It must refer to ‘iterating’ over the string itself, i.e., by unpacking it into three different variables, notwithstanding that each condition is mutually exclusive of the other two.

In turn, then, we must ignore that Entry.title carries the value and type of str. Instead, it is an Entry object, or a single attribute of a single Entry object. It makes sense that the appeal to ‘a value of str’ is not helpful, since that can be true of any use of the alphabet. Put another way, entry.title is first and foremost an Entry object. It is only incidentally, and of necessity, a ‘str.’

Does that mean that we can salvage this operation by calling str() on the attribute? A string is clearly iterable. But in this context, it is not a string. It is an attribute of Entry communicated to humans by means of a string. That makes sense in light of the common use of str in models. We take a representation and convert it to a ‘true’ string so we can understand and make use of it. If that’s the reasoning, then calling str() on it should make this problem go away.

No. I still have exactly the same error on exactly the same Entry object. Neither my try/except nor my exclude() have been given any effect.

t.title is not iterable, even if its string value is. Therefore my conditional variable assignments are ineffective, and might for convenience be considered None. Therefore, Entry.save(t) is also None, because nothing has changed. Therefore, when we hit the except clause, nothing has changed. We are still presenting Django with a title that already exists, thus triggering the traceback.

But if Entry.save(t) is a nullity, how can Django be presented with an integrity violation? This entry object has a title. It is not None. If all the intervening statements are None, then Entry.save(t) is also None, and None is not an integrity error. If this field on the model can’t be None, we still have an error, but it isn’t from duplicating an existing title. None != a pre-existing title.

exclude(**kwargs)
Returns a new QuerySet containing objects that do not match the given lookup parameters.

I put in a bunch of prints:

def doubledateqs_noklass():
    """for using regex with querysets
    SEE https://docs.djangoproject.com/en/5.0/ref/models/querysets/#regex and
    https://forum.djangoproject.com/t/filter-using-regex-doesnt-return-results-but-printing-query-in-db-does/27323"""

```snip'''
    dd_title = eoa.filter(title__regex=r'\d{4}-\d{2}-\d{2}\s\d{4}-\d{2}-\d{2}')
    for t in dd_title:
        if str(t) == "2024-02-10 2023-06-06 ToT GDE insert":
            dd_title = dd_title.exclude(t)
             # According to the docs, there is no need to re-assign the name,
             # because this is what exclude() does anyway.
            continue


        # The 'a' condition (equality) is not met by this instance
        # but it is met by the b condition:


        # these are full objects, not just the title attribute
        print(f"This is t: {t}") # This is t: 2024-02-10 2023-06-06 ToT GDE insert
        print(F"The type of t is: {type(t)}") # The type of t is: <class 'ktab.models.Entry'>
        print(F"dd_title is of type: {type(dd_title)}") # dd_title is of type: <class 'django.db.models.query.QuerySet'>
        te = str(t.title)
        print(F"The value of te is: {te}") # The value of te is: 2024-02-10 2023-06-06 ToT GDE insert
        print(f"The type of te is: {type(te)}") # The type of te is: <class 'str'>


            except (psycopg2.errors.UniqueViolation, django.db.utils.IntegrityError):
                dd_title.exclude(t)
            continue
        elif te[:10] < te[11:21]:
            c = te[:10] + te[22:]
            t.title = c
            try:
                Entry.save(t)
            except (psycopg2.errors.UniqueViolation, django.db.utils.IntegrityError):
                dd_title.exclude(t)
            continue

But this time, Django went straight to the non iterable objection, and dropped all reference to integrity errors:

/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/bin/python /Users/malikarumi/Projects/snippets/dubble.py
Traceback (most recent call last):
  File "/Users/malikarumi/Projects/snippets/dubble.py", line 386, in <module>
    doubledateqs_noklass()
  File "/Users/malikarumi/Projects/snippets/dubble.py", line 195, in doubledateqs_noklass
    dd_title.exclude(t)
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/query.py", line 982, in exclude
    return self._filter_or_exclude(True, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/query.py", line 992, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/query.py", line 997, in _filter_or_exclude_inplace
    self._query.add_q(~Q(*args, **kwargs))
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1375, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1396, in _add_q
    child_clause, needed_inner = self.build_filter(
                                 ^^^^^^^^^^^^^^^^^^
  File "/Users/malikarumi/Library/Caches/pypoetry/virtualenvs/chronicle-YMNQB2Tm-py3.12/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1268, in build_filter
    arg, value = filter_expr
    ^^^^^^^^^^
TypeError: cannot unpack non-iterable Entry object

We have no integrity errors now. Instead, we have gone straight to non-iterable Entry object. How? Why? Did ‘continue’ have no effect? Because if it did, the following prints would have told us that t was now a different object. But that didn’t happen, nor did any integrity errors.
How? Python has not yet gotten to my ‘unpacking’ variable assignments. Does .exclude(t) require iteration? But it’s a queryset. Querysets were made for iteration.

2

What is eoa here?

Note: This:

does not modify dd_title. It’s an expression that returns a new queryset - but you are not saving that queryset. From the perspective of what you’re doing, this has no effect.

You would need to write this as dd_title = dd_title.exclude(...) (see next point)

The exclude method on a queryset is expecting lookup parameters as the argument, not an instance of the model. (See exclude()) You would need to write this as something like dd_title = dd_title.exclude(id=t.id)

Now, even if you make these changes, I’m not sure it’s going to have the effect in the for loop that I believe you want it to have. (Note: It’s considered extremely bad form within python to add or remove items from the list being iterated over.) I would suggest you make a copy of that queryset object and alter the copy in the loop and not the “base” version.

Side note:

What is Entry at this point? If it’s the name of the model, then this is an unusual phrasing of this expression. It’s typically written in Django as t.save(). (Given the metaprogramming involved with models, it wouldn’t surprise me to find out that this doesn’t even work correctly when written this way, but that’s mostly unfounded conjecture on my part with no direct knowlege that there is anything wrong with doing it this way. It’s simply not how it is shown in any of the examples in the docs.)

3

you chnage object name that t. so, you save object name that t.
but you save the model class.

for t in dd_title:
    # these are full objects, not just the title attribute
    te = t.title
    if te[:10] == te[11:21]:
        a = te[11:]
        t.title = a
        try:
            # Entry.save(t)
            t.save()
        except (psycopg2.errors.UniqueViolation, django.db.utils.IntegrityError):
            dd_title.exclude(t)
        continue
4

In the general case, this is not a problem. By passing the instance as the first parameter, it’s actually syntactically correct. Assuming there’s nothing special about the class, this notation does work. (Try it!) The potential here is not so much the syntax of the statement itself, but whether there are any attributes on the class created by the metaclass that could be causing a problem.