makemigrations throwing error if the model has query filter().exist()

[mubashir-v]

Continuing the discussion from error - no such table:

Hi guys,
I encountered a problem creating migration files (py manage.py makemigrations)
Error message : django.db.utils.OperationalError: no such table: app_newcollection
And Error is throwing from view.py file
Here is the code structure which I used

model.py

class NewCollection(models.Model):
      product = models.OneToOneField(Products, on_delete=models.CASCADE)
      endDate=models.DateTimeField(auto_now=True)

view.py

class NewCollectionViewset(viewsets.ModelViewSet):
    productIds=[]
    if NewCollection.objects.filter().exists():
        newcollection=NewCollection.objects.all()
        for new in newcollection:
            productIds.append(new.product_id)
    queryset = Products.objects.filter(id__in=productIds)
    serializer_class = productSerializer

Observation

class NewCollectionViewset(viewsets.ModelViewSet):
    productIds=[]
    #if NewCollection.objects.filter().exists():
        #newcollection=NewCollection.objects.all()
        #for new in newcollection:
            #productIds.append(new.product_id)
    queryset = Products.objects.filter(id__in=productIds)
    serializer_class = productSerializer

Migrations apply if I comment out above lines.
(Exception was throwing from this line → NewCollection.objects.filter().exists())

Does this mean that the new models cannot be filtered from view.py initially?

Github Link : GitHub - mubashir-cvr/Ecommerce-Django
(Dependancies are available in requirements.txt)

[KenWhitesell]

Have you done any rearranging, reorganization, or refactoring of your code, models, apps, or views?

The error message you have posted says:

@mubashir-v

django.db.utils.OperationalError: no such table: app_newcollection

But the code in the repo you shared shows:

class ApisConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'apis'

which means the table name should be apis_newcollection.

Also, your repo that you shared doesn’t include any of the existing migration files.

What is the output from a manage.py showmigrations?

It might also be helpful if you posted the complete traceback. The error shown here might not be the root cause - there may be a more fundamental issue that needs to be addressed.

[mubashir-v]

Hi ,
manage.py showmigrations? also throwing same error.

@KenWhitesell

Also, your repo that you shared doesn’t include any of the existing migration files.

I have Untracked migration files using git ignore.

@KenWhitesell

which means the table name should be apis_newcollection .

This should create on makemigrations right?

What I am understood is - view.py file expecting apis_newcollection table from particular line I mentioned above .
But comic fact is this :- The exception/error is throwing in the operation of creating (makemigrations) apis_newcollection table

[KenWhitesell]

Try starting from a fresh database. Drop and recreate your database and start from scratch.

@mubashir-v

I have Untracked migration files using git ignore.

That’s a particularly bad idea. Migration files are as much a part of your project as your models.py and views.py. This point alone leads me to believe that your migration structure is messed up beyond repair.

Try from a fresh database and see what happens.

@KenWhitesell

Have you done any rearranging, reorganization, or refactoring of your code, models, apps, or views?

And you didn’t answer this question - which gets to the point of why it’s looking for the table named app_newcollection.

[mubashir-v]

Thanks @KenWhitesell

@KenWhitesell

Have you done any rearranging, reorganization, or refactoring of your code, models, apps, or views?

No.
I noticed this while deploying project in server
steps I followed →
→ Deleted migration files (Always I followed the procedure by deleting the migration files and deploying them to the server and doing migration from there. May be I am wrong here.)
→ Encountered error as mentioned.
→ Comment out view.py file.
→ Migration successful.
→ Uncommented view.py file.
→ All functionality working fine.
→ Then I took freez > requirmensts.txt
→ Tried in different machine
→ Same error encountered there.
I can’t add logic filter().exist() inside view.py file before apply migration thats the issue.

[KenWhitesell]

@mubashir-v

→ Deleted migration files (Always I followed the procedure by deleting the migration files and deploying them to the server and doing migration from there. May be I am wrong here.)

Yep, that’s the problem. Your migrations setup is broken.

Never delete a migration file unless you’re going to be rebuilding your database from scratch.

It’s possible to recover from deleted migrations, but it tends to involve a lot of manual effort.

Drop and recreate your database and start again.

Never delete a migrations file unless you know exactly what you are doing and what the implications are from doing it.

[mubashir-v]

OKey ,
I was doing wrong till now then
So I need to make sure apply migrations before going to write view.py file for the new model right?

[KenWhitesell]

You need to drop and recreate the database, then do your makemigrations.

[mubashir-v]

@KenWhitesell

You need to drop and recreate the database, then do your makemigrations.

You mean cleaning database ? That I did (deleted sqlite in dev env) still no luck .

System is trying to execute NewCollection.objects.filter().exists() while making makemigrations call where NewCollection is not created yet.

@KenWhitesell

Never delete a migrations

Still confusion here . What if I have multiple people working on the same project while tracking migration files in repo?. It will cause a conflict in migrate right?

[KenWhitesell]

@mubashir-v

You mean cleaning database ? That I did (deleted sqlite in dev env) still no luck .

And the makemigrations is failing in your dev env?

Please provide the full traceback of the error you receive.

Regardless, that’s not going to help in your other environments. All of your databases should be dropped and recreated. You have completely broken your migrations environment. (You can save your application data and restore it, but unless you’re willing to spend the time to understand enough of the Django internals to know how to recover from this, your fastest path to recovery is a complete from-scratch rebuild of the database.)

@mubashir-v

Still confusion here . What if I have multiple people working on the same project while tracking migration files in repo?. It will cause a conflict in migrate right?

It can. There are ways to manage / mitigate this. See Migrations | Django documentation | Django to get some ideas.

We used a slightly different technique for one of our projects. We had a “staging” branch that is “pre-main”. Developers were free to make model changes, etc and migrations in their own branches. Those migrations didn’t get checked in. Their code got merged into “staging”. Then makemigrations was run in the “staging” branch to create the “official” migrations. Those migrations were committed back into the repo before being merged to “main”.
(It worked, but turned out to be more effort than it was worth in the long run and so we never adopted it again.)

[mubashir-v]

(env) mubashir@mubi-mac-air Ecommerce-Django % python manage.py makemigrations
/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1416: RuntimeWarning: DateTimeField Products.created_date received a naive datetime (2021-12-24 21:23:34.921370) while time zone support is active.
 warnings.warn("DateTimeField %s received a naive datetime (%s)"
Traceback (most recent call last):
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
   return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: apis_newcollection

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/manage.py", line 22, in <module>
   main()
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/manage.py", line 18, in main
   execute_from_command_line(sys.argv)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
   utility.execute()
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
   self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
   self.execute(*args, **cmd_options)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/management/base.py", line 393, in execute
   self.check()
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/management/base.py", line 419, in check
   all_issues = checks.run_checks(
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
   new_errors = check(app_configs=app_configs, databases=databases)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config
   return check_resolver(resolver)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver
   return check_method()
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/urls/resolvers.py", line 412, in check
   for pattern in self.url_patterns:
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
   res = instance.__dict__[self.name] = self.func(instance)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/urls/resolvers.py", line 598, in url_patterns
   patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
   res = instance.__dict__[self.name] = self.func(instance)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/urls/resolvers.py", line 591, in urlconf_module
   return import_module(self.urlconf_name)
 File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
   return _bootstrap._gcd_import(name[level:], package, level)
 File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
 File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
 File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 850, in exec_module
 File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/core/urls.py", line 8, in <module>
   path('api/', include('apis.urls')),
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/urls/conf.py", line 34, in include
   urlconf_module = import_module(urlconf_module)
 File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
   return _bootstrap._gcd_import(name[level:], package, level)
 File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
 File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
 File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 850, in exec_module
 File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/apis/urls.py", line 12, in <module>
   from .views import *
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/apis/views.py", line 71, in <module>
   class NewCollectionViewset(viewsets.ModelViewSet):
 File "/Users/mubashir/DEV/VIOUI/Ecommerce-Django/apis/views.py", line 74, in NewCollectionViewset
   if NewCollection.objects.filter().exists():
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/models/query.py", line 808, in exists
   return self.query.has_results(using=self.db)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/models/sql/query.py", line 552, in has_results
   return compiler.has_results()
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1145, in has_results
   return bool(self.execute_sql(SINGLE))
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
   cursor.execute(sql, params)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
   return super().execute(sql, params)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
   return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
   return executor(sql, params, many, context)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
   raise dj_exc_value.with_traceback(traceback) from exc_value
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
 File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
   return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: apis_newcollection```

[mubashir-v]

class OffersaleViewset(viewsets.ModelViewSet):
    # define queryset
    
    queryset = Products.objects.filter(offers__offerPrice__gt=0).exists() //for test
    # specify serializer to bce used
    serializer_class = productSerializer

Replicated .filter().exists() query in view.py for different model (products) same error

File "/Users/mubashir/DEV/VIOUI/env/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: apis_products
(env) mubashir@mubi-mac-air Ecommerce-Django % 

[KenWhitesell]

Ok, so after taking a more detailed look at the code, I do see another set of problems.

You’re trying to execute queries within the class definition - that didn’t register to me the first time I looked at it.

There’s a huge difference between:
newcollection=NewCollection.objects.all()
which creates a queryset but does not try to resolve it, and:
NewCollection.objects.filter().exists() which does try to resolve the queryset by executing the query.

This latter statement does not belong in a class definition at the class level. Keep in mind that classes are defined once - that statement would only ever be executed when the module is first loaded, not on every request.

When you’re needing to perform this type of logic within a generic class, it needs to be placed in the appropriate function overridden by your class definition. (It works in the same way as the Django-provided generic class-based views.)
In this specific case, you probably want this logic placed in the get_queryset method of the Viewset class, not at the module layer.

See Viewsets - Django REST framework for more information.

[mubashir-v]

@KenWhitesell

NewCollection.objects.filter().exists() which does try to resolve the queryset by executing the query.

Got it .
I will try to wrap those logics inside functions / get_queryset.
Thanks a lot for looking into this and guide me