Server 500 Error while trying to view models

Hi all,

This is an abridged version of a more detailed post that was blocked by the auto-mod.
I will provide as much detail as I can.
I’m afraid I will spend another hour writing a post that gets blocked. (If a mod reads this I would really love to have that post, even if it were just in an email :grin: pls )

I know this is not a perfect setup. I explained motivations and other considerations in my blocked post.
My issue is that after this hodgepodge, I log into the django admin site click on the model, it returns a 500 error.

My setup

curl → pandas → sqlalchemy → postgresql ← → dajngo ← → uwsgi ← → nginx

Everything looks good with postgres as far as I can tell. Django can read the db,
I can query the db from the shell.

python manage.py inspectdb >> app/models.py

Above command produces a relevant file. I played with the config a bit, details in blocked post. No luck.

In the admin site, I can add new records. I think these are committed to the db, I need to check again (sorry).
Not really sure where to go from here.

By the way I dont really know any sql beyond basic CRUD, I actually picked postgres because of its transactional logic. So the first three entries in the chain above are my quick and dirty method to getting something that works.

One detail I almost forgot. The table is created from pandas through SQLAlchemy. I’m sure I could work the “curl and pandas” bits into my django project, that just feels like admitting defeat at the present moment.

I can provide more details if you want to know. I hope I was clear enough.

Any help would be greatly appreciated.

Thanks!

First, I can confirm that what you’re trying to do is quite doable - we do it all the time. (Accessing databases and tables that are created / managed by other systems. There’s no need to change anything in Django to use that data. There are some limitations in what you can do, but it all can be made to work.)

For further assistance, we’re going to need to see the model and the full traceback you’re getting from the 500. We’ll probably also need to see the database table you’re trying access, along with other details around your database configuration. (Specifically, is the database you’re trying to access this way the same database you’re using as your Django database? That adds some other potential complications with configuration.)

Note: when posting code or errors here, please enclose it between lines of three backtick - ` characters. This means you’ll have a line of ```, then your code, then another line of ```.

Second note: For the purpose of this thread, we probably don’t need to see all the tables at first. Pick one table that is demonstrating this issue and we’ll get that fixed first.

Thanks so much for your reply.

I am not quite sure how to get the logs with DEBUG=False. But I can set it to True if needed.
Looking at django/utils/log.py I see that it should be emailed to the site admin. Do I just need to give my django superuser an email address? I always leave it blank. I’m going to try that, if it works I’ll post the logs. If the logging needs to be configured, I may need some pointers.

99.9% sure I have everything is the same database. I’m guessing it would be a good idea to split them up then. That really makes a lot more sense for my potential use use cases.

Here is the database. 2nd entry was the very first table I named. I ended up using maintable and records. which are identical only differ by a single entry.
My thinking was to have sqlalchemy create a new table when records were updated. I see how rudimentary my approach was. I should just update my single table and include dataetime info in pandas.

                       List of relations
 Schema |               Name                |   Type   | Owner  
--------+-----------------------------------+----------+--------
 public | Main_Table                        | table    | robot
 public | Mon May 24 19:27:49 2021          | table    | robot
 public | auth_group                        | table    | robot
 public | auth_group_id_seq                 | sequence | robot
 public | auth_group_permissions            | table    | robot
 public | auth_group_permissions_id_seq     | sequence | robot
 public | auth_permission                   | table    | robot
 public | auth_permission_id_seq            | sequence | robot
 public | auth_user                         | table    | robot
 public | auth_user_groups                  | table    | robot
 public | auth_user_groups_id_seq           | sequence | robot
 public | auth_user_id_seq                  | sequence | robot
 public | auth_user_user_permissions        | table    | robot
 public | auth_user_user_permissions_id_seq | sequence | robot
 public | django_admin_log                  | table    | robot
 public | django_admin_log_id_seq           | sequence | robot
 public | django_content_type               | table    | robot
 public | django_content_type_id_seq        | sequence | robot
 public | django_migrations                 | table    | robot
 public | django_migrations_id_seq          | sequence | robot
 public | django_session                    | table    | robot
 public | main_table                        | table    | robot
 public | maintable                         | table    | robot
 public | records                           | table    | robot
(24 rows)

Here is the table, named records

        Ids        | Last Name | First Name |        Email        |    Areas     
-------------------+-----------+------------+---------------------+--------------
 recCZ8yDCyg3O0OCg | Dow       | John       | JD@jj.jj            | Integrations
 recGs9mNWjMidlG6S | Jenkins   | Leroy      | lj@ww.lko           | Accounting
 reclfZIhteIA4ipYQ | Doe       | Jane       | anotheremail@jfj.jj | Recruiting

Check syslog or your uwsgi log for the stack trace, it may be available there.

If that’s the actual name of the table, it’s quite possible that Django is going to have problems with that. (I don’t know right off-hand if the ORM has problems with spaces in table names.)

We’ll also need to see the model used for that table, along with the actual schema definition for it.

Sorry I wasn’t clear. I left everything as is in the table. The one you are referring to was the very first table I made.
I forget the error it gave, but you are correct and that failed miserably. I couldn’t even figure out how to query it in postgres.
I have that table unregistered from the admin site and deleted from models.py.

Also after looking through every log I could find, I re-read the doc page about logging and found how to write the log to a file. Will update with that soon. But I think my database schema may be an issue. I never configured one. So new tables default to the public schema.

models.py (in its present state)

from django.db import models   
class Records(models.Model):                                                                                                                                                                  
    ids = models.TextField(db_column='Ids', blank=True, null=True)  # Field name made lowercase.                                                                                              
    last_name = models.CharField(max_length=100,db_column='Last Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                    
    first_name = models.CharField(max_length=100,db_column='First Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                  
    email = models.EmailField(max_length=100,db_column='Email', blank=True, null=True)  # Field name made lowercase.                                                                          
    areas = models.TextField(db_column='Areas', blank=True, null=True)  # Field name made lowercase.                                                                                          
                                                                                                                                                                                              
    class Meta:                                                                                                                                                                               
        managed = True                                                                                                                                                                        
        db_table = 'records'                                                                                                                                                                                   

from the postgres cli

\dpublic.records

                                             Access privileges
 Schema |               Name                |   Type   |   Access privileges   | Column privileges | Policies 
--------+-----------------------------------+----------+-----------------------+-------------------+----------
 public | Main_Table                        | table    |                       |                   | 
 public | Mon May 24 19:27:49 2021          | table    |                       |                   | 
 public | auth_group                        | table    |                       |                   | 
 public | auth_group_id_seq                 | sequence |                       |                   | 
 public | auth_group_permissions            | table    |                       |                   | 
 public | auth_group_permissions_id_seq     | sequence |                       |                   | 
 public | auth_permission                   | table    |                       |                   | 
 public | auth_permission_id_seq            | sequence |                       |                   | 
 public | auth_user                         | table    |                       |                   | 
 public | auth_user_groups                  | table    |                       |                   | 
 public | auth_user_groups_id_seq           | sequence |                       |                   | 
 public | auth_user_id_seq                  | sequence |                       |                   | 
 public | auth_user_user_permissions        | table    |                       |                   | 
 public | auth_user_user_permissions_id_seq | sequence |                       |                   | 
 public | django_admin_log                  | table    |                       |                   | 
 public | django_admin_log_id_seq           | sequence |                       |                   | 
 public | django_content_type               | table    |                       |                   | 
 public | django_content_type_id_seq        | sequence |                       |                   | 
 public | django_migrations                 | table    |                       |                   | 
 public | django_migrations_id_seq          | sequence |                       |                   | 
 public | django_session                    | table    |                       |                   | 
 public | main_table                        | table    |                       |                   | 
 public | maintable                         | table    | robot=arwdDxt/arobot |                   | 
 public | records                           | table    |                       |                   | 
(24 rows)

It does look like robot Doesnt have access privileges on “records”. To be clear, I had similar results with maintable

models.py (“maintable” version)

class Maintable(models.Model):                                                                                                                                                                
    ids = models.TextField(db_column='Ids', blank=True, null=True)  # Field name made lowercase.                                                                                              
    last_name = models.TextField(db_column='Last Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                                   
    first_name = models.TextField(db_column='First Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                                 
    email = models.TextField(db_column='Email', blank=True, null=True)  # Field name made lowercase.                                                                                          
    areas = models.TextField(db_column='Areas', blank=True, null=True)  # Field name made lowercase.                                                                                          
                                                                                                                                                                                              
    class Meta:                                                                                                                                                                               
        managed = False                                                                                                                                                                       
        db_table = 'maintable'                                                                                                                                                                

Neither of the models you’ve posted (Records, Maintable) have a primary key defined in those models. As a result, Django is going to assume that there’s a field named id that is the primary key. Since there doesn’t appear to be a column by that name in the table fragment you displayed, the admin will throw an error when trying to retrieve rows from those tables.
(You can work with “foreign tables” that don’t have a primary key in your own code, but generally speaking, not with the admin.)

Yes! I was wondering about the primary key, I knew something had to be happening behind the scenes but I forgot to look into it.

The stacktrace starts with an error about ‘subtitle’. Is that related to the ‘id’?

Exception while resolving variable 'subtitle' in template 'admin/login.html'.
Traceback (most recent call last):
  File "/home/robot/menv/lib/python3.9/site-packages/django/template/base.py", line 829, in _resolve_lookup
    current = current[bit]
  File "/home/robot/menv/lib/python3.9/site-packages/django/template/context.py", line 83, in __getitem__
    raise KeyError(key)
KeyError: 'subtitle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/robot/menv/lib/python3.9/site-packages/django/template/base.py", line 835, in _resolve_lookup
    if isinstance(current, BaseContext) and getattr(type(current), bit):
AttributeError: type object 'RequestContext' has no attribute 'subtitle'

This line further down is exactly what you are referring to.

django.db.utils.ProgrammingError: column records.id does not exist
LINE 1: SELECT "records"."id", "records"."Ids", "records"."Last Name...

I can post the whole stack trace if you would like. But it is quite long. I will add primary keys and see how it goes

Final Update. A working model.py.
I ended up simply naming the index in the pandas dataframe “id”. For today, this is perfect.
Email validations working. Django is awesome :slight_smile:
I learned a lot here. Thanks Ken!

# Feel free to rename the models, but don't rename db_table values or field names.                                                                                                            
from django.db import models                                                                                                                                                                  
                                                                                                                                                                                              
class Newrecord(models.Model):                                                                                                                                                                
                                                                                                                           
    last_name = models.CharField(max_length=100, db_column='Last Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                   
    first_name = models.TextField(max_length=100,db_column='First Name', blank=True, null=True)  # Field name made lowercase. Field renamed to remove unsuitable characters.                  
    email = models.EmailField(max_length=100,db_column='Email', blank=True, null=True)  # Field name made lowercase.                                                                          
    areas = models.CharField(max_length=100,db_column='Areas', blank=True, null=True)  # Field name made lowercase.                                                                           
    ids = models.CharField(max_length=100,db_column='Ids', blank=True, null=True,unique=True)  # Field name made lowercase.                                                                   
                                                                                                                                                                                              
    class Meta:                                                                                                                                                                               
        managed = False                                                                                                                                                                       
        db_table = 'newrecord'