GET requests not updating on page reload

Using Django Forms & APIs
1

Hi, folks, I have a question and I’m not sure where else to go to research the solution.
I’ve got a small project where I’ve been messing around with webscrapers and search engines.
From the home page if a search term is typed in it loads a results page with 2 columns.
Left column are just the results from larger search engines (Google, yahoo, etc). The right column will be internal results. The code in views.py that handles that is as follows → 

engine = SearchEngine()

def search(request):
    search_string = request.GET.get('search-for')
    search_results = engine.google(search_string)

    template = 'searchnet/results.html'
    context = {
        "location": location,
        "weather": weather,
        "menu": MAIN_MENU,  
        "search_results": search_results[:10],
        "side_results": search_string[::-1],
    }

    return render(request, template, context)

index.html and results.html are the same except for underneath the search bar in results.html is the 2 column results table I described above.
The Django HTML for that is → 

{% block front %}
    <table class="results-table col-12 py-4">
        <thead>
            <tr>
                <th><h4>Mainstream: </h4></th>
                <th><h4>Sidestream: </h4></th>
            </tr>
        </thead>
        <tbody>
            {% for result in search_results %} 
            <tr class="results-row">
                <td class="mainstream-data">
                    <div class="mainstream-content">
                        <p><a href="{{ result.url }}">{{ result.url }}</a></p>
                        <a href="{{ result.url }}">{{ result.title }}</a>
                        <p class="result-description">{{ result.description }}</p>
                    </div>
                </td> 
                <td class="sidestream-data"><a href="">{{ side_results }}</a></td> 
            </tr>
            <tr class="spacer-row"></tr>
            {% endfor %}
        </tbody>
    </table>
    
{% endblock front %}

My urls.py is → 

app_name = 'searchnet'

urlpatterns = [
    path('', views.index, name='index'),
    path('search/', views.search, name='search'),
    path('category/<str:category_name>/', views.category, name='category'),

    # path('results/<str:search_string>/', views.results, name='results'),
]

Both the search forms from index.html and results.html are exactly the same → 

{% block search %}
    <form action="{% url 'searchnet:search' %}" method="GET" class="col-12 col-lg-auto mb-3 mb-lg-0 me-lg-3">
        <input type="search" name="search-for" class="form-control" placeholder="Search.. " aria-label="Search">
    </form>
{% endblock search %}

My issue is: When a search term is entered on index.html it loads the results.html with the formatted results properly (so far).
But to make another search query from results.html is different. The left side doesn’t reload the new search term and displays results from the previous search from index.html. The right side is only setup to display the search term backwards right now, and it does that with the new term, but the left side doesn’t pick up the new search query.
Even if I go back to index.html, reload the page and type in another query results.html will still show the pre-reload query on the left. Even the url changes to the new search term, but the left side doesn’t reload and update it.

What is it I need to look at to solve this issue? Thanks

2

What you’ve posted obviously isn’t the complete view. I can’t see how you’re getting the behavior described from what you’ve posted, so I think we may need to see the entire view. We may end up needing to see the base template being used as well, but I’m not sure about that yet.

Also, what is this “SearchEngine” object you’re using? Where does that come from? (Is that something you’ve written or a 3rd party library?)

3

Ok, I can do that. I was hoping if it was a generic issue that what I posted would be enough; if not, no worries…
here is the entire views.py file → 

from django.shortcuts import render
from django.http import HttpResponse, Http404, HttpResponseRedirect
from .apis.weather.weather import location, weather
from .apis.news.news import articles
from .classes.search import SearchEngine
# from .classes.search import search_query
from .config import MAIN_MENU


engine = SearchEngine()

def index(request): 
    template = 'searchnet/index.html'
    context = {
        "location": location,
        "weather": weather,
        "menu": MAIN_MENU,
     
        }
    return render(request, template, context)


def search(request):
    search_string = request.GET.get('search-for')
    search_results = engine.google(search_string)

    template = 'searchnet/results.html'
    context = {
        "location": location,
        "weather": weather,
        "menu": MAIN_MENU,  
        "search_results": search_results[:10],
        "side_results": search_string[::-1],
    }

    # return HttpResponseRedirect(reverse('searchnet:results', args=(search_string,)))
    return render(request, template, context)

def category(request, category_name):
    template = 'searchnet/category.html'
    context = {
        "location": location,
        "weather": weather,
        "menu": MAIN_MENU,  
        "content": category_name,
    }
    return render(request, template, context)

Here’s the entire urls.py → 

from django.urls import path
from . import views


app_name = 'searchnet'

urlpatterns = [
    path('', views.index, name='index'),
    path('search/', views.search, name='search'),
    path('category/<str:category_name>/', views.category, name='category'),
]

Here’s results.html → 

{% extends 'searchnet/base.html' %}


<!-- weather -->
{% block weather %}
<div class="text-color-third">{{ weather }} in {{ location }}</div>
{% endblock weather %}

<!-- register -->
{% block register %}
<a class="btn btn-sm btn-outline-secondary" href="#">Register</a>
{% endblock register %}

<!-- menu -->
{% block menu %}
<div class="nav-scroller py-1 mb-2">
    <nav class="nav d-flex justify-content-between">
        {% for category in menu %}
        <a class="p-2 text-color-primary" href="{% url 'searchnet:category' category %}">{{ category }}</a>
        {% endfor %}
    </nav>
</div>
{% endblock menu %}

<!-- search -->
{% block search %}
    <form action="{% url 'searchnet:search' %}" method="GET" class="col-12 col-lg-auto mb-3 mb-lg-0 me-lg-3">
        <input type="search" name="search-for" class="form-control" placeholder="Search.. " aria-label="Search">
    </form>
{% endblock search %}

<!-- results -->
{% block front %}
    <table class="results-table col-12 py-4">
        <thead>
            <tr>
                <th><h4>Mainstream: </h4></th>
                <th><h4>Sidestream: </h4></th>
            </tr>
        </thead>
        <tbody>
            {% for result in search_results %} 
            <tr class="results-row">
                <td class="mainstream-data">
                    <div class="mainstream-content">
                        <p><a href="{{ result.url }}">{{ result.url }}</a></p>
                        <a href="{{ result.url }}">{{ result.title }}</a>
                        <p class="result-description">{{ result.description }}</p>
                    </div>
                </td> 
                <td class="sidestream-data"><a href="">{{ side_results }}</a></td> 
            </tr>
            <tr class="spacer-row"></tr>
            {% endfor %}
        </tbody>
    </table>
    
{% endblock front %}

and index.html is exactly the same as results.html except the front block, which is → 

{% block front %}
<div class="content-sidebar my-5">

    <div class="row col-9">
        <div class="row">
            <div class="col-6 p-0">
                <img src="{{ articles.2.urlToImage }}" alt="image pertaining to article" width=460 height=275><br />    
            </div>
            <div class="col-4 p-2 bg-light">
                <a href="{{ articles.2.url }}"><h5 class="text-color-black text-center">
                    {{ articles.2.title }}
                </h5></a>
                <a href="{{ articles.2.url }}">
                    <p class="text-size-16 text-center">{{ articles.2.description|slice:":160" }}...[To Source]</p> 
                </a>
            </div>
        </div>

        <div class="row">
            {% for article in articles|slice:"11:16" %}
            <div class="card col-2 p-0" style="width: 153" >
                <img class="card-img-top" src="{{ article.urlToImage }}" alt="Card image cap" width=153 height=76>
                <div class="card-body">
                <a href="{{ article.url }}"><h6 class="card-title text-size-14 text-weight-bold text-color-black text-center" data-bs-toggle="tooltip" title="{{ article.title }}">{{ article.title|slice:":45" }}...</h6></a>
                </div>
            </div>
            {% endfor %}
        </div>
    </div> 

    <div>
        <iframe width="auto" height="235" src="https://maphub.net/embed/188169?button=0&panel=1&panel_closed=1" frameborder="0"></iframe>
        <div class="feed-buttons my-4">
            <a href="#"><button type="button" name="reddit-button" class="btn reddit-color mx-2 px-2"><i class="fab fa-reddit" aria-hidden="true"></i>Reddit</button></a>
            <a href="#"><button type="button" name="twitter-button" class="btn twitter-color mx-2 px-2"><i class="fa-brands fa-twitter" aria-hidden="true"></i>Twitter</button></a>
            <a href="#"><button type="button" name="rss-button" class="btn rss-color mx-2 px-2"><i class="fa-solid fa-rss" aria-hidden="true"></i>&nbsp;RSS</button></a>

        </div>
    </div>

</div>





<h3>Off Track</h3>
<div class="row align-items-start py-5">
    
    {% for lower_article in lower_articles|slice:"0:8" %}
        <div class="row col-9">
            {% comment %} <img class="mr-3" src="..." alt="Generic placeholder image"> {% endcomment %}
            <div class="media-body border-left border-dark m-1 py-1 px-2"><a href="{{ lower_article.url }}">
            <h5 class="mt-0">{{ lower_article.title }}</h5></a>
            </div><br>
        </div>
    {% endfor %}

</div>

{% endblock front %}

I hope this clarifies it more, thanks for your response

EDIT: I just saw the part where you asked about SearchEngine object. That’s a personal class not a 3rd party one. In the interest of completion it looks like this → 

from bs4 import BeautifulSoup
import metadata_parser
from requests import get
from urllib.parse import quote
from typing import List, Dict


class SearchEngine:
    def __init__(self):
        self.results = []

    def google(self, query: str, max_results = 10, lang = "en", **proxies) -> List[Dict[str, str]]:
        engine = get(f"https://www.google.com/search?q={quote(query, safe='')}&num={max_results}&hl={lang}", headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) ''Chrome/61.0.3163.100 Safari/537.36'}, proxies = {}).text
        part = BeautifulSoup(engine, "lxml").find_all("div", attrs={"class": "yuRUbf"})

        return self.to_list(part)
...

    def to_list(self, part):
        """
        Generates a list of dictionaries for each result
        """
        for i in range(len(part)):
            title = part[i].find("h3").text
            url = part[i].find("a", href = True)["href"]
            description = SearchEngine.get_meta(url)
            
            self.results.append({"title": title, "url": url, "description": description})
            # self.results.append({"title": title, "url": url})
        
        return self.results

    @staticmethod
    def get_meta(url):
        page = metadata_parser.MetadataParser(url)
        meta = page.metadata['meta']
        try:
            description =  f"{meta['description'][:60]}..."
        except KeyError:
            description = "There is no description for this page"

        return description

query = "jupiter"
def main():
    engine = SearchEngine()
    results = engine.google(query, localhost=3000)

    # meta = engine.get_meta(results[0]['url'])
    # print(f"description: {meta}")

    # print(results)

    for result in results:
        # print(f"site_meta description: {engine.get_meta(result['url'])}\n")
        print(result)

    # print(__import__('sys').version_info)

if __name__ == '__main__':
    main()
4

There’s nothing apparent to me yet.

I know you’ve said that it’s not updating the page - have you determined whether it’s issuing the query to google?

If I were trying to debug this, I’d be tracking this down step by step. That probably means either running this in a debugger or starting to add some targeted print statements to verify that things are happening as expected in the right places.

5

I think I see it …

You’re initializing the results once, when the class is instantiated.

When is that done? At module load time. It’s never reinitialized.

In the google method, you have:

which calls the to_list method which has:

So it’s appending the new results to the end of the existing list. You’re not seeing the new results because you’re limiting the display to the first 10 entries of the list when the new ones are being added past that.

You may want to reinitialize your list at the beginning of to_list.

Along the same lines - I hope you’re not planning to have more than one person use this at a time. If you do, you’re going to encounter problems.

You don’t want to create your instance at the module level. You want to create your instance of the engine within your view. If you’re going to store data within the instance, you need a new instance each time the view is called.

1 Like
6

I was away a few days …
Thank you. Moving engine = SearchEngine() inside the view fixed my problem and your explanation made perfect sense