Pagination not working for displaying search results

Hi, when I try to display my search results from elasticsearch using pagination, it does not work. I suspect the problem is that my view doesn’t accept GET methods, also maybe the way I call upon the elasticsearch client to get my results. Below is my code. Any help is appreciated!

views.py:

def search(request):
    # page_number = request.GET.get('page', 1)
    results_per_page = 20  # You can adjust this number
    # If this is a POST request we need to process the form data
    if request.method == "POST":
if request.method == "POST":
        # Create a form instance and populate it with data from the request:
        form = SearchForm(request.POST)
        # check whether it's valid:
        if form.is_valid():
            # Get query from form.cleaned_data
            query = form.cleaned_data['query']
            print(query)
            # Start ES client
            client = Elasticsearch(ENDPOINT, api_key=API_KEY, ca_certs=CERT)
            resp = client.search(index= INDEX, size=100, query={ "match": { "userId": query }})

            # If match query has no hits/results, try wildcard query
            if(resp['hits']['total']['value'] == 0):
                formatted_hits = []
                print("No results found")
            else:
                hits = resp['hits']['hits']
                # print(hits)
                
                formatted_hits = []
                #Format hits to get desired fields and store them in array
                for hit in hits:
                    formatted_hit = {
                        'datetime': hit['_source']['datetime'],
                        'brand': hit['_source']['brand'],
                        'userId': hit['_source']['userId'],
                        'role': hit['_source']['role'],
                        'request': hit['_source']['request'],
                    }
                    formatted_hits.append(formatted_hit)

                paginator = Paginator(formatted_hits, results_per_page)

                page_number = request.GET.get('page', 1)
                page_obj = paginator.get_page(page_number)

                context = {
                'hits': formatted_hits,
                'form': form,
                'page_obj': page_obj,
                'query': query,
                'total_hits': len(formatted_hits)
                }

                # return results html with formatted_hits array and form
                return render(request, 'audit_logs/results.html', context)
    # if a GET (or any other method) we'll create a blank form
    else:
        form = SearchForm()

    return render(request, "audit_logs/search.html", {"form": form})

templates:
search.html:

<h1>You are searching for</h1>
<form action="" method="post">
    {% csrf_token %}
    {{ form }}    
    <input type="submit" value="Search">
</form>

results.html

<h1>You are searching for</h1>
<form action="" method="get">
    {% csrf_token %}
    {{ form }}    
    <input type="submit" value="Search">
</form>
<h2>Results:</h2>

<!-- Debug info -->
<p>Debug info:</p>
<ul>
    <li>Total hits: {{ total_hits }}</li>
    <li>Number of pages: {{ page_obj.paginator.num_pages }}</li>
    <li>Current page: {{ page_obj.number }}</li>
    <li>Has next page: {{ page_obj.has_next }}</li>
    <li>Has previous page: {{ page_obj.has_previous }}</li>
</ul>

{% if hits %}
    <table class="result">
        <thead>
            <tr>
                <th>Datetime</th>
                <th>Brand</th>
                <th>userId</th>
                <th>role</th>
                <th>request</th>
            </tr>
        </thead>
    {% for hit in page_obj %}
            <tbody>
                <tr>
                    <td>{{ hit.datetime }}</td>
                    <td>{{ hit.brand }}</td>
                    <td>{{ hit.userId }}</td>
                    <td>{{ hit.role }}</td>
                    <td>{{ hit.request }}</td>
                </tr>
            </tbody>
    {% endfor %}
</table>

<!-- Pagination -->
<div class="pagination">
    <span class="step-links">
        {% if page_obj.has_previous %}
            <a href="?page=1">&laquo; first</a>
            <a href="?page={{ page_obj.previous_page_number }}">previous</a>
        {% endif %}

        <span class="current">
            Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}.
        </span>

        {% if page_obj.has_next %}
            <a href="?page={{ page_obj.next_page_number }}">next</a>
            <a href="?page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
        {% endif %}
    </span>
</div>
{% elif hits is not None %}
    <p class="no-results">No results found.</p>
{% else %}
    <p>No results found</p>
{% endif %}

Welcome @TomZ !

Please provide a more detailed description of the issue you are facing beyond saying:

How is it not working? Are you getting any error messages? (Console or browser?) Are you not getting any results? Are you getting the wrong results? Is the pagination not working? What output are you seeing?

1 Like
  • Your search view should handle both POST requests (for the initial search) and GET requests (for pagination).

  • Currently, the view only processes POST requests to perform the search. For pagination, you need to ensure the view can handle GET requests and maintain the search query.

  • When paginating, you need to ensure the query parameter is passed along with the page number.

Try something like this:

from django.core.paginator import Paginator
from django.shortcuts import render
from elasticsearch import Elasticsearch
from .forms import SearchForm  # Adjust the import based on your project structure

def search(request):
    results_per_page = 20  # Adjust this number as needed
    query = request.GET.get('query', '')  # Get the query from GET parameters for pagination

    if request.method == "POST":
        # Create a form instance and populate it with data from the request:
        form = SearchForm(request.POST)
        if form.is_valid():
            query = form.cleaned_data['query']
            # Start ES client
            client = Elasticsearch(ENDPOINT, api_key=API_KEY, ca_certs=CERT)
            resp = client.search(index=INDEX, size=100, query={"match": {"userId": query}})

            # If match query has no hits/results, try wildcard query
            if resp['hits']['total']['value'] == 0:
                formatted_hits = []
                print("No results found")
            else:
                hits = resp['hits']['hits']
                formatted_hits = [
                    {
                        'datetime': hit['_source']['datetime'],
                        'brand': hit['_source']['brand'],
                        'userId': hit['_source']['userId'],
                        'role': hit['_source']['role'],
                        'request': hit['_source']['request'],
                    }
                    for hit in hits
                ]

            paginator = Paginator(formatted_hits, results_per_page)
            page_number = request.GET.get('page', 1)
            page_obj = paginator.get_page(page_number)

            context = {
                'hits': formatted_hits,
                'form': form,
                'page_obj': page_obj,
                'query': query,
                'total_hits': len(formatted_hits)
            }

            return render(request, 'audit_logs/results.html', context)
    else:
        form = SearchForm()

        if query:  # Handle GET requests for pagination
            client = Elasticsearch(ENDPOINT, api_key=API_KEY, ca_certs=CERT)
            resp = client.search(index=INDEX, size=100, query={"match": {"userId": query}})

            if resp['hits']['total']['value'] == 0:
                formatted_hits = []
                print("No results found")
            else:
                hits = resp['hits']['hits']
                formatted_hits = [
                    {
                        'datetime': hit['_source']['datetime'],
                        'brand': hit['_source']['brand'],
                        'userId': hit['_source']['userId'],
                        'role': hit['_source']['role'],
                        'request': hit['_source']['request'],
                    }
                    for hit in hits
                ]

            paginator = Paginator(formatted_hits, results_per_page)
            page_number = request.GET.get('page', 1)
            page_obj = paginator.get_page(page_number)

            context = {
                'hits': formatted_hits,
                'form': form,
                'page_obj': page_obj,
                'query': query,
                'total_hits': len(formatted_hits)
            }

            return render(request, 'audit_logs/results.html', context)

    return render(request, "audit_logs/search.html", {"form": form})

Then, ensure the form in results.html maintains the query parameter when paginating:

<form action="" method="get">
    {{ form.as_p }}
    <input type="hidden" name="query" value="{{ query }}">
    <input type="submit" value="Search">
</form>

Also, ensure that the pagination links maintain the query parameter:

<div class="pagination">
    <span class="step-links">
        {% if page_obj.has_previous %}
            <a href="?query={{ query }}&page=1">&laquo; first</a>
            <a href="?query={{ query }}&page={{ page_obj.previous_page_number }}">previous</a>
        {% endif %}

        <span class="current">
            Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}.
        </span>

        {% if page_obj.has_next %}
            <a href="?query={{ query }}&page={{ page_obj.next_page_number }}">next</a>
            <a href="?query={{ query }}&page={{ page_obj.paginator.num_pages }}">last &raquo;</a>
        {% endif %}
    </span>
</div>

1 Like

Hi, thank you for the reply!

What doesn’t work is that when I click next or last, which are the HTML paginator elements, the correct search results disappear. I tried reading the Paginator documentation on Django and look for similar problems on stack overflow, but couldn’t find much. I am not getting any error messages

That is why I suspect my view not taking Get requests and the way I handle the query is wrong, as I might not be passing it when I click next to see the next results.

Hi, thank you for your reply!

Will give this a try and get back to you if this works. Thank you!

Hi @anefta , your solution works for me and the pages are working correctly now. Thank you very much!

The error was that my view didn’t handle GET requests and that the query parameter was not passed along.

However, the problem I am getting now is that after a search, when I try to search again with a new query, the previous query is still being searched for instead of the new one.

(http://127.0.0.1:8000/audit_logs/search/?query=querryNew&query=queryOld)

^ This is what I am getting in the link.

This could be happening because results.html has the method get now.

<form action="" method="get">
    {{ form.as_p }}
    <input type="hidden" name="query" value="{{ query }}">
    <input type="submit" value="Search">
</form>

When we do a search after a previous search, we are on results.html. Since it is a get method, it doesn’t trigger if request.method == "POST", and doesn’t replace the query here query = form.cleaned_data['query'] in the search view

Also, when we paginate, are we doing the search again every single time? If we are, then that might get very expensive if the size was to increase.

I might want to implement some sort of slicing method using the elasticsearch client, so I only need to query and search the portion I want whenever I go to a new page.

Thank you again for your help!

1 Like

A search view would usually only use GET requests, not POSTs. Changing your view to use GET will probably greatly simplify things and make it more likely pagination will work.

if method is get, form request action url without parameter.

so, you should:

<a href={url}></a>

or

<form action="{url}" method="get">
    {input name page}
    {{ form.as_p }}
    <input type="hidden" name="query" value="{{ query }}">
    <input type="submit" value="Search">
</form>

Hi, thank you for the reply.

I will try to see if just using GET method works and get back to you.

Hi, thank you for the reply.

Form actions is something I still don’t fully understand yet as I am new to Django. For the url parameter, should it be pointing to the results url? However, the current way I have implemented it is that there is no results URL, just a template to display search results.

I don’t know what you’re talking about.
What I’m saying is that the url requested in the next two is the same.

<form action="aaa.com/?s=2&d=5" method="get">
    ...
</form>
<form action="aaa.com/" method="get">
    ...
</form>

Oh ok, I think I understand what you mean

The form action is the URL that the form should submit to. If it’s left blank, or omitted, then the form submits to the same URL as the current one, the page that you’re on. So your form should probably start something like:

<form action="{% url 'search' %}" method="get">

if your search() view is mapped to a URL called search. Talking of which:

The URL is the one that maps to your search() view.

Regarding this:

I suspect you have two query input tags in your form.

We haven’t seen your SearchForm class, but does it have a query field, in which the user types their search query? If so, that is output in the template when you do {{ form.as_p }}. But you’ve also put this in your form’s HTML:

<input type="hidden" name="query" value="{{ query }}">

Which is adding a second (hidden) query input with a value of the previous query. Remove that hidden field from the template.

BTW, Django 5.1 will be released very shortly and includes the new querystring template tag which is ideal for using with the pagination links. See the note here: Django 5.1 release notes - UNDER DEVELOPMENT | Django documentation | Django

1 Like

Yes, you are completely right. My search form looks like this:

from django import forms

class SearchForm(forms.Form):
    query = forms.CharField(label='Search userId', max_length=100)

By just removing the hidden input in the results.html template, the search now works as expected (new search queries are made without the old query in there as well). The pagination still works as well.

I tried adding the URL into the form action for the search.html and results.html template, I get the error below. Is it because they are part of the app audit_logs and not the root.

Reverse for 'search' not found. 'search' is not a valid view function or pattern name.

This is what my root folder urls.py looks like:

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("audit_logs/", include("audit_logs.urls")),    #add our app into the urlconfig
    path('admin/', admin.site.urls),
]

What my app urls.py looks like:

from django.urls import path

from . import views

app_name = "audit_logs"
urlpatterns = [
    path("", views.index, name="index"),
    path("search/", views.search, name="search"),
]

I might stick to using POST method for searches and GET methods for the pagination for now as it works but will report here what happens if I do change it.

Thank you for your help Phil!

Looks like the URL for the form needs to be the "search" URL that’s in the "audit_logs" app. So:

<form action="{% url 'audit_logs:search' %}" method="get">
1 Like

Will give this a try, thank you!