listing items with filtering/sorting do not work with pagination

Hello to all the good people in the forum. Hope you are doing well.
Im a django/python student (by myself, don’t know anyone to feedback or share the learning)

VIEW 1
I have a link in my webapp that returns a template filled with a list of items from the DB (always 17.000 rows aprox) so the view use pagination to show 30 per page (prevent browser for loading all at once)

@login_required()
def connected_list(request):
    
    template = "template.html"   
    sql = "base sql query like select * from"
    
    with connections['BD'].cursor() as cursor:
        cursor.execute(sql)
        rows = cursor.fetchall()       
        registers = []   
        for row in rows:
            register = {}

            register['field1'] = row[0]
            register['field2'] = row[1]
            register['field3'] = row[2]            
        

            registers.append(register)

    total = len(registers)
    p = Paginator(registers, 30)
    page = request.GET.get('page')
    conjunto = p.get_page(page)
    
    context = {"total" : total,
            "conjunto": conjunto,
            }               
    return render(request, template, context)

VIEW 2
The template also have a search bar and a selection for different sorting criteria both inside one form
which send this 2 parameters via POST request (to keep the url nice and not showing the parameters in the url). This form calls a 2nd view for doing another query to the DB with the filtering parameter and/or the sorting parameter and returns it to the same template from the first view and also has pagination by 30 items per page.

@login_required()
def search_connected_list(request):
    template = "template.html"
    sql = "base sql query like select * from"

      #empieza prueba
    sorts = ["sort_by_field_1", "sort_by_field_1", "sort_by_field_1"]

    search = request.POST.get('searched')
    if search == None:
        search = ''
    if (search != None) and (search != ''):
              
        sql += f" adds more filtering with the variable {search}%')"
        
    sort = request.POST.get('orden')
    
    if sort in sorts:
        sql += f" ORDER BY {sort} ASC"
        
    #termina prueba
    with connections['BD'].cursor() as cursor:
        cursor.execute(sql)
        rows = cursor.fetchall()       
        registers = []   
        for row in rows:
	    register = {}

            register['field1'] = row[0]
            register['field2'] = row[1]
            register['field3'] = row[2]  					    
                     
            registers.append(register)

    total = len(registers)
    p = Paginator(registers, 30)
    page = request.GET.get('page')
    conjunto = p.get_page(page)
    
    context = {"total" : total,
            "conjunto": conjunto,
            "search" : search,
            # "orden" : sort
            }
                 
    return render(request, template, context)

PROBLEM
pagination dont work when I call the 2nd view, the list of rows returned by the view is ok (filter criteria and sorting, at least the first 30 I see in the template) but when I click on “go to next page” (GET request) suddenly Im in the page 2 (app dont crash) but from the total number of items (like the base query calling all the items in view 1). For example, I filter using a word and get 6000 rows, going to page 2 I see “page 2 from 567” and total rows show 17.000 instead of “page 2 of 200” and total of 6000.

THE TEMPLATE (just in case)

{% extends "base.html" %}
{% load static %}
{% block extra_js %}
<script src="{% static "js/main_jadmin.js"%}"></script>
{% endblock %} 
{% block content %} 
{% if conjunto %}
<div class="container-fluid mt-5 mb-5">
    <div>
        <p>Conectados: {{total}}</p> 
    </div>
    <div>
        <form action="{% url 'jadmin:busqueda' %}" method="POST">
            {% csrf_token %}
            <input type="search" class="" name="searched" value={{search}}>
            <label for="orden">ordenar por:</label>
            <select id="orden" name="orden">
                <option value="" selected></option>
                <option value="username">Usuario</option>
                <option value="acctstarttime">Inicio de sesion</option>
            </select>
            <button class="btn btn-outline-secondary" type="submit">Buscar</button>      
        </form> 
    </div>
   
    <div class="row">
        <div class="col-sm-12 col-md-12 col-lg-12 col-xl-12">
            <div class="table-responsive">
                <table id="datatables_conected" class="table table-striped">
                    <thead>
                        <tr>
                            <th class="centered">Usuario</th>
                            <th class="centered">Inicio de sesión</th>
                            <th class="centered">Duración</th>
                            <th class="centered">Agregador</th>
                            <th class="centered">VLAN</th>
                            <th class="centered">IPv4</th>
                            <th class="centered">IPv6</th>
                            <th class="centered">Consumo Download</th>
                            <th class="centered">Consumo Upload</th>
                            <th class="centered">Mac cliente</th>
                            <th class="centered">Acciones</th>
                        </tr>
                    </thead>
                    <tbody>
                        {% for register in conjunto %}                 
                        <tr>
                            <td>{{register.username}}</td>
                            <td>{{register.start_time|date:"Y-m-d H:i:s"}}</td>
                            <td>{{register.duration}}</td>
                            <td>{{register.nas_ip}}</td>
                            <td>{{register.vlan}}</td>
                            <td>{{register.ipv4}}</td>
                            <td>{{register.ipv6}}</td>
                            <td>{{register.down_usage}}</td>
                            <td>{{register.up_usage}}</td>
                            <td>{{register.mac}}</td>
                            <td>
                                <div class="button-row">
                                    <a href="#"><button type="submit" class="btn btn-warning"><i class="fa-solid fa-pencil"></i></button></a>
                                    <a href="#"><button type="submit" class="btn btn-danger"><i class="fa-solid fa-trash-can"></i></button></a>
                                </div>
                            </td>
                        </tr>                    
                        {% endfor %}
                    </tbody>
                </table>
                        {%comment%} paginacion  {% if search %}searched={{search}}&{%endif%} {%endcomment%}
                        {% if conjunto.has_previous %}
	                    <a href="?page=1">Primera</a>
	                    <a href="?page={{conjunto.previous_page_number}}">Anterior</a>
                        {% endif %}

                        pagina {{conjunto.number}} de {{conjunto.paginator.num_pages}}

                        {% if conjunto.has_next %}
	                    <a href="?page={{conjunto.next_page_number}}">Siguiente</a>
	                    <a href="?page={{conjunto.paginator.num_pages}}">Ultima</a>
                        {% endif %}
                        {%comment%} paginacion {%endcomment%}
{% else %}

                <p>No hay registros disponibles</p>

{% endif %}                 
            </div>
        </div>
    </div>
</div>
{% endblock %}

any ideas?! Thanks in advance

First, you should focus on learning how to use the orm to make your queries. You absolutely do not want to build the queries manually, and especially not using the method you have constructed here. (You’ve created a very serious and dangerous SQL-injection vulnerability in your code.)

You imply that you’re relatively new to this - have you worked your way through the official Django tutorial? If you haven’t, that’s the place to start. If you have, then you want to go back and review the queries you had constructed in the exercises.

To the core of your question, you will need to alter your template to include the query filters in the urls being rendered by the pagination links.

First, thank you for your response Ken.

About django doc, yes I read it…just it does not have all the knowledge, is like a beginer’s guide.

About raw SQL, im doing it because Im working with a legacy database (also the app will be only in a private lan so it would not be exposed to the internet). I know it is recommended to use “inspectdb” command to make a model from legacy DB but i’m afraid to do that because Im not sure if the table will be modified in some way “accidentally” using the migrate or makemigrations commands. My main goal is to be 100% sure not modifying the table because that DB is in production and it has a lot of sensible data. So my decision of using raw SQL was born from the fear of breaking my DB. I know is not the best practice u.u

Finally, about altering the template to include filters in the url…are you refering to use GET request and past the parameters in the url right ? because using POST request in the form makes the parameters go behind the curtain and not in the url. Do you know any example or guide to work with this case? I just need a push to understand this.

The tutorial is exactly that - a beginners guide. Once you’ve worked your way through it, then you’re usually in a position to start digging through the full docs - which do have “all the knowledge”.

That’s covered quite well at How to integrate Django with a legacy database | Django documentation | Django

Yes. The easiest way to make this work is to use a GET to submit the query filters. (It’s also more in line with the general HTTP philosophy in that the submission of query parameters is an idempotent operation on that model - it’s not going to alter the database.)

good morning !

Ok, I will run some test today about using inspectdb on a test mysql DB and will do some migration to the model to check if it’s reliable.

Concerning the parameters sending, ok I get the easiest way is by GET but never seen an example of using the POST way. Do you know any site or reference? My goal is not make url’s so ugly. Maybe is there a way to clean the url’s later ?

That shouldn’t be your goal. Your goal should be to create a usable system that behaves the way people expected it to - both your users and developers who may also end up working on it.

Probably the most significant benefit of get-based filter parameters from the user’s perspective is that they can be bookmarked - which means they can be reused or shared. Supporting that fundamental functionality is far more important a UI/UX issue than any perceived esthetics regarding the appearance of a url.

Hello Ken, thanks for your replies, they mean a lot.
[/quote]
Probably the most significant benefit of get-based filter parameters from the user’s perspective is that they can be bookmarked - which means they can be reused or shared.
[/quote]
I had not thought of that, so I will do that.

Regarding avoiding raw SQL, I already read all the docs refering to work with legacy db’s and inspectdb command but there is something that is not clear so I tested:

1 - I created a test database in the MySQL server with only one table called “Alumnos” and fill it with some rows of data.
2 - I added the dictionary parameters for the connection in the settings.py
3 - I run the “inspectdb” command for that DB and obtain a model and pasted it into models.py of a test app created only to test this. I used the “managed = False” flag (according to docs the database would be safe and not taking changes from migrations)
4 - configured a router module to map the test app with the test mysql DB
5 - Run makemigration and migrate commands to see. (py manage.py migrate --database=test_db)
6 - Enter mysql engine via terminal and PUM, my table name was renamed from “alumnos” to “testapp_alumnos” (it appended the app name before the table name) and added all the auth app tables and all the django app tables and all the models from the other apps that I made before this inside this test_db.

wtf why django is doing this, I am missing something ? I swear I read all the docs related to this.

Please post the DATABASES from the settings.py file. Also please post the Alumnos model.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    },
    'freeradius':{
        'ENGINE':'django.db.backends.mysql',
        'NAME':'base1',
        'USER':'pepe',
        'PASSWORD':'aaa',
        'HOST':'192.168.53.11',
        'PORT':'3306'
    },
      'radius':{
        'ENGINE':'django.db.backends.mysql',
        'NAME':'base2',
        'USER':'pepe',
        'PASSWORD':'bbb',
        'HOST':'192.168.53.11',
        'PORT':'3306'
    },
      'CoopVGG':{
        'ENGINE':'django.db.backends.mysql',
        'NAME':'base3',
        'USER':'pepe',
        'PASSWORD':'ccc',
        'HOST':'192.168.53.11',
        'PORT':'3306'
    },
    'raddb':{
        'ENGINE':'django.db.backends.mysql',
        'NAME':'base4',
        'USER':'pepe',
        'PASSWORD':'ddd',
        'HOST':'192.168.53.11',
        'PORT':'3306'

    },
    'test_db':{
        'ENGINE':'django.db.backends.mysql',
        'NAME':'KeyManager',
        'USER':'',
        'PASSWORD':'',
        'HOST':'localhost',
        'PORT':'3306'
    }

}

DATABASES_ROUTERS = ['db_routers/routers.py/ProbandoRouter',]

1 -I use the default DB for a single model in a single app which has the django and auth tables also
2 -The test_db use to try this is the last one with the user and pass left blank
3 - All the other mysql DB’s between are the ones that I have in production at work so I was doing the connections via raw SQL just in case

Alumnos model:

from django.db import models

class Alumnos(models.Model):
    name = models.CharField(max_length=50, blank=True, null=True)
    surname = models.CharField(max_length=50, blank=True, null=True)
    age = models.IntegerField(blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'alumnos'
        #app_label = 'probando'

Router for test_db (just pasted the docs example and edited):

class ProbandoRouter:
    """
    A router to control all database operations on models in the
    auth and contenttypes applications.
    """
    route_app_labels = {'probando'}

    def db_for_read(self, model, **hints):
        """
        Attempts to read auth and contenttypes models go to auth_db.
        """
        if model._meta.app_label in self.route_app_labels:
            return 'test_db'
        return None

    def db_for_write(self, model, **hints):
        """
        Attempts to write auth and contenttypes models go to auth_db.
        """
        if model._meta.app_label in self.route_app_labels:
            return 'test_db'
        return None

    def allow_relation(self, obj1, obj2, **hints):
        """
        Allow relations if a model in the auth or contenttypes apps is
        involved.
        """
        if (
            obj1._meta.app_label in self.route_app_labels or
            obj2._meta.app_label in self.route_app_labels
        ):
           return True
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        """
        Make sure the auth and contenttypes apps only appear in the
        'auth_db' database.
        """
        if app_label in self.route_app_labels:
            return db == 'test_db'
        return None

You do have an issue with your model definition - You don’t have a primary key defined. See point #1 in the docs for managed at Model Meta options | Django documentation | Django

I’m not sure that explains everything, but it’s a start.

Also, I’m a bit confused about a couple things.

You wrote:

which leads me to believe the app name is testapp, but your router is defined with

which means the routers aren’t going to interfere with an app named testapp.

If you don’t want any migration operation to be performed on a database, you want to return False in allow_migrate for that database.

See Multiple databases | Django documentation | Django

(Actually, you probably want to review that entire section of the docs with the intent of understanding exactly what that router example is doing in each function, and why.)

Sorry, when I said “testapp” earlier I was refering that I started an app to test, but the name that I used is “probando” (testing in spanish). So the reference in the router is ok.

Tomorrow I will try to do all again but with the “allow_migrate” on False and see what happen. I will update.

Hello again, sorry for the delay.

Finally did it !, the routers.py approach was not working so what Im doing now for using the ORM is using the “using()” method on all the queries like this:

alumno = Alumnos.objects.using(‘test_db’).get(id=pk)

this way I can create the models using “inspectdb” and then use them by this method correspondig each model to a database of my preference.

Thank you for the help Ken hope you are doing well and have a nice week