Use data returned from external API to create a query of a model

Hello, I am extremely new to Django and trying to learn on my own. I am making progress slowly, but I am stuck on one scenario and not sure if it can be accomplished or not.

I currently call 2 separate external APIs for different purposes:

1st purpose - search a site based on a search term the user provides. This is built into the form itself for the search and basically returns a list of news articles:

forms.py

def search(self):
        result = {}
        search_term = self.cleaned_data['topic']
        api_key = settings.NEWS_API
        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
        url = endpoint.format(api_key=api_key, search_term=search_term)        
        response = requests.get(url)
        if response.status_code == 200:
            result = response.json()
            result['success'] = True
        else:
            result['success'] = False
            if response.status_code == 404:
                result['message'] = 'No entry found for "%s"' % search_term
            else:
                result['message'] = 'The News API is not available at the moment'
        return result

2nd purpose - call an external API to populate data in a model called ‘Bias’. This is done via a script. This data is informational data about various news sources:

get_bias.py

class Command(BaseCommand):
    def handle(self, *args, **options):        
        url = "https://media-bias-fact-check-ratings-api2.p.rapidapi.com/fetch-data"

        headers = {
           "x-rapidapi-key": "xxxxxxxxxxxx",
           "x-rapidapi-host": "media-bias-fact-check-ratings-api2.p.rapidapi.com"
           }
       
        response = requests.get(url, headers=headers)
        decoded_response = response.content.decode('utf-8-sig')
        data = json.loads(decoded_response)
        if response.status_code == 200:
           for i in data:
               Bias.objects.update_or_create(
                   mbfc_id = i['Source ID#'],
                   defaults = {
                       'source_name': i['Source'],
                       'mbfc_url': i['MBFC URL'],
                       'bias': i['Bias'],
                       'factual': i['Factual Reporting'],
                       'type': i['Media Type'],
                       'source_url': i['Source URL'],
                       'credibility': i['Credibility'],
                       'country': i['Country'],
                   }
               )
           self.stdout.write(self.style.SUCCESS('Successful'))
        else:
            self.stdout.write(self.style.ERROR('Failed'))
          

Here is what I am trying to accomplish:

In the first search one of the fields returned in the API is called “Source”. I want to take that field that is returned and use it to search against the model ‘Bias’.

Ultimately my goal is to show all of the fields of the first “search” API and also include several fields from the model data but only IF the source is found in the model.

I have searched everything I can find but no luck - I took a stab at this myself and it’s not working.

Here is my view and what I have tried:

views.py

def news_view(request):        
        topic_search = {}
        bias_data = Bias.objects.all().values()
        if 'topic' in request.GET:
                form = NewsForm(request.GET)
                if form.is_valid():
                        topic_search = form.search()
        else:
                form = NewsForm()
        context = {
                'topic_search': topic_search,
                'bias_data': bias_data, 
                'form': form
        }
        return render(request, 'news/news_api.html', context=context)

news_api.html

 {% if topic_search %}
            <h2>results:</h2>
        
            {% if topic_search.success %}
                {% for result in topic_search.data %}
                    
                        <h3>{{ result.topic }}</h3><br>

                        <div class="card text-center">
                            <h3>{{ result.title }}</h3><br>
                            <p>{{ result.description }}</p><br>
                            <p>{{ result.published_at }}</p><br>
                            <p>{{ result.source }}</p><br>  
                            {% if result.source == bias_data.source_url %}
                                <p>Bias: {{ bias_data.bias }}</p>
                            {% endif %}
                            <a href="{{ result.url }}">{{ result.url }}</a>
                        </div>
                    
                {% endfor %}        
            {% else %}
                <p><em>{{ search_result.message }}</em></p>

Any help would be appreciated!!

I’m not sure I’m following what you’re trying to describe here.

The way I’m reading this, you’re using the requests module to call an external API, which is returning a JSON response. In that response is a key named “Source”, that you want to use in some type of query on the Bias model.

If this is a correct interpretation, it would help if you posted the Bias model, and show a sample of what you might retrieve from the API and how that relates to what you want to search for in Bias. It is always easier to answer questions like this using real(ish) data than trying to address everything in the abstract.

(If this is not a correct interpretation, then please clarify what it is you are asking for.)

Thank you for responding - yes that is the correct interpretation of what I am trying to do. Here is my Bias model. When I get the data from the first API there is a ‘domain’ field (sorry I called it source in the first explanation) that matches the ‘source_url’ field of this bias model:

class Bias(models.Model):
    source_name = models.CharField(max_length=200)
    mbfc_url = models.URLField()
    bias = models.CharField(max_length=100)
    country = models.CharField(max_length=100)
    factual = models.TextField()
    type = models.TextField()
    credibility = models.TextField()
    source_url = models.URLField(null=True, blank=True)
    mbfc_id = models.IntegerField()

    def __str__(self):
        return self.source_name

Here is an example of the data being returned from the first API call:

{
            "source_id": "theatlantic.com-1",
            "domain": "theatlantic.com",
            "language": "en",
            "locale": "us",
            "categories": [
                "politics"
            ]
        },

Here is the data that gets loaded into the Bias model from the 2nd api call:

Source:"(The) Atlantic"
MBFC URL:"https://mediabiasfactcheck.com/the-atlantic/"
Bias:"Left-Center"
Country:"USA"
Factual Reporting:"High"
Media Type:"Magazine"
Source URL:"theatlantic.com"
Credibility:"High"
Source ID#:102697

“domain” == “source url”

Hope this helps clarify…

Ok, so to relate this back to your original question, are you asking how to write a query on the Bias model that looks for all rows where source_url equals the domain field of the API response?

Yes, I believe so. I am displaying the article data that is returned from the first API, and I want to incorporate a few of the fields from the bias model into that view by pulling the information for the domain for each result…

Sorry if I am overexplaining…but here is how my results from the first API search are set up (for now) - all of the fields currently showing are from the first API call which gets news articles.

I want to query the Bias model based on the source of each article (highlighted yellow) and return some fields from the model and show them along with the other information… the red area. I just can’t figure out how to get the source of each article from the Bias model…

I tried just adding a query in my view to return all bias records - and then added an if statement in the template to compare the article domain (source) to the bias source_url and if it matches then display the data…that did not work.

Again to rephrase for my understanding - what you want is:

  1. Call the API
  2. Use the results of the API to query the Bias model
  3. Render a template containing both the data from the API call and the Bias model
    (Please clarify if this is not correct.)

What is the basic way to query a model? (Did you work your way through the official Django tutorial? If so, then you may want to review what you had done in part 2.)

Yes your understanding it correct.

I have gone through the tutorial and I do understand the general way to query a model. I think where I am struggling still though is how to utilize the data from the article API as search parameter in the model query. I just can’t wrap my head around how to get the specific source of each article and use that to query the bias model for that source.

I have tried the following, but it doesn’t work:

in my views.py file:

def news_view(request):        
        topic_search = {} 
        
        if 'topic' in request.GET:
                form = NewsForm(request.GET)
                if form.is_valid():
                        topic_search = form.search()
                        domain_list = topic_search.get('domain', []) 
                        bias_data = Bias.objects.filter(source_url__in=domain_list)                       
        else:
                form = NewsForm()
   
        return render(request, 'news/news_api.html', {'form': form, 'topic_search': topic_search, 'bias_data': bias_data})

I added that bias_data query thinking I could then use that in my template:

 {% if topic_search %}
            <h2>results:</h2>
        
            {% if topic_search.success %}
                {% for result in topic_search.data %}
                    
                        <h3>{{ result.topic }}</h3><br>

                        <div class="card text-center">
                            <h3>{{ result.title }}</h3><br>
                            <p>{{ result.description }}</p><br>
                            <p>{{ result.published_at }}</p><br>
                            <p>{{ result.source }}</p><br>  
                            <p>Bias: {{ bias_data.bias }}</p>
                            
                            <a href="{{ result.url }}" target="_blank">{{ result.url }}</a>
                        </div>
                    
                {% endfor %}        
            {% else %}
                <p><em>{{ search_result.message }}</em></p>

            {% endif %}

Based upon the clarifications above, I’m not sure I’m seeing where a form factors into this, unless the form is providing parameters for the API call.

The sequence of events in your view would be exactly what I’ve outlined above.

  1. Call the API
  2. Use the API-retrieved data in a query
  3. Render a template with the data retrieved above.

Where does a form fit into this sequence?

Well the way I structured this is the API call for the articles is happening in the form…it works but maybe that was not the best way to do this.

forms.py

class NewsForm(forms.Form):
    topic = forms.CharField(label="search", max_length=200)

    def search(self):
        result = {}
        search_term = self.cleaned_data['topic']
        api_key = settings.NEWS_API
        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
        url = endpoint.format(api_key=api_key, search_term=search_term)        
        response = requests.get(url)
        if response.status_code == 200:
            result = response.json()
            result['success'] = True
        else:
            result['success'] = False
            if response.status_code == 404:
                result['message'] = 'No entry found for "%s"' % search_term
            else:
                result['message'] = 'The News API is not available at the moment'
        return result

You’re right, it’s not.

Use the form to accept input from the user and validate it.

Do the work in the view. The three steps outlined above should all either be in the view or in functions directly called by it.

Does topic_search.get('domain') return a list or a single element?

Your code implies that it’s returning a list, but your example above shows it returning a string.

Let me start with restructuring the code to use the form as it should be and go from there. I think that is where I keep getting stuck is how I pass that information into the view when I have pulled it in the form. Again, I am very new at this so I am probably not using exactly the right language either, so I am working on that. Thanks, I will do some work and see if I can get closer.

Well I moved the api call for the articles to the view but I am still just not getting how to get the one field from the returned data and use it to search the Bias model and return that data for use in the template. This is what I have now:

views.py

def news_view(request):          
        search_term = None 
        if request.method == 'GET':
                form = NewsForm(request.GET)
                if form.is_valid():  
                                                    
                        search_term = form.cleaned_data['topic']
                        api_key = settings.NEWS_API
                        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
                        url = endpoint.format(api_key=api_key, search_term=search_term) 

                        try:
                                response = requests.get(url)
                                response.raise_for_status()     
                                response_data = response.json() 
                                filter_values = [data['source'] for data in response_data['data']] 
                        except requests.exceptions.RequestException as e:
                                response_data = {'error': str(e)}
                else:
                        form = NewsForm()  

                queryset = Bias.objects.filter(source_url__in=filter_values)        

                return render(request, 'news/news_api.html', {'form': form, 'response_data': response_data, 'queryset': queryset})

template

{% if response_data %}
                {% for data in response_data.data %}
                    
                        <h3>{{ data.topic }}</h3><br>

                        <div class="card text-center">
                            <h3>{{ data.title }}</h3><br>
                            <p>{{ data.description }}</p><br>
                            <p>{{ data.published_at }}</p><br>
                            <p>{{ data.source }}</p><br>  
                            {% if queryset.source_url == data.source %}
                            <p>Source Bias: {{ queryset.source_name }}</p>
                            {% endif %}
                            
                            <a href="{{ data.url }}" target="_blank">{{ data.url }}</a>
                        </div>
                    
                {% endfor %}        
            {% else %}
                <p><em>{{ data.message }}</em></p>

            {% endif %}

The code looks good, so the next issue is to check out the data.

You have:

This implies that whatever you’re storing in that field must be a url. (e.g. https://example.com) A bare domain name (example.com) does not pass the validation rules for this field.

If the entries in source_url contain values like: http://example.com, then your filter_values would need to be something like ['http://test.com', 'http://example.com'] and not ['test.com', 'example.com']

(It might be worth adding a print(filter_values) line before this one to verify what it is you’re actually searching for. You might also want to verify that the data in the source_url field is correct and is what you expect.)

Well I thought I had it… it worked once…but anytime I exit the app for some reason when I try to run it again I get this message so there is still something wrong in my code somewhere:

Request Method:	GET
Request URL:	http://127.0.0.1:8000/news/
Django Version:	4.2.20
Exception Type:	UnboundLocalError
Exception Value:	
local variable 'response_data' referenced before assignment
Exception Location:	C:\Users\kpopo\VWS\vws_site\news\views.py, line 34, in news_view
Raised during:	news.views.news_view
Python Executable:	C:\Users\kpopo\VWS\Scripts\python.exe
Python Version:	3.10.3
Python Path:	
['C:\\Users\\kpopo\\VWS\\vws_site',
 'C:\\Python310\\python310.zip',
 'C:\\Python310\\DLLs',
 'C:\\Python310\\lib',
 'C:\\Python310',
 'C:\\Users\\kpopo\\VWS',
 'C:\\Users\\kpopo\\VWS\\lib\\site-packages']

If you’re looking for assistance with this, please post the current version of the view along with identifying which line is line 34 of that file.

It was the same as I posted before but first I received this error…so I moved the queryset to inside the try clause. Once I did that I then received the next error…

Line 32 is the queryset line

C:\Users\kpopo\VWS\vws_site\news\views.py, line 32, in news_view
                queryset = Bias.objects.filter(source_url__in=filter_values)           …
Local vars
def news_view(request):          
        search_term = None 
        if request.method == 'GET':
                form = NewsForm(request.GET)
                if form.is_valid():  
                                                    
                        search_term = form.cleaned_data['topic']
                        api_key = settings.NEWS_API
                        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
                        url = endpoint.format(api_key=api_key, search_term=search_term) 

                        try:
                                response = requests.get(url)
                                response.raise_for_status()     
                                response_data = response.json()
                                filter_values = [data['source'] for data in response_data['data']]
                                 
                        except requests.exceptions.RequestException as e:
                                response_data = {'error': str(e)}
                else:
                        form = NewsForm()

                queryset = Bias.objects.filter(source_url__in=filter_values)          

                return render(request, 'news/news_api.html', {'form': form, 'response_data': response_data, 'queryset': queryset})
C:\Users\kpopo\VWS\vws_site\news\views.py, line 33, in news_view
                return render(request, 'news/news_api.html', {'form': form, 'response_data': response_data, 'queryset': queryset}) …
Local vars

Updated view that caused this one, line 33 is the last one:

def news_view(request):          
        search_term = None 
        if request.method == 'GET':
                form = NewsForm(request.GET)
                if form.is_valid():  
                                                    
                        search_term = form.cleaned_data['topic']
                        api_key = settings.NEWS_API
                        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
                        url = endpoint.format(api_key=api_key, search_term=search_term) 

                        try:
                                response = requests.get(url)
                                response.raise_for_status()     
                                response_data = response.json()
                                filter_values = [data['source'] for data in response_data['data']]
                                queryset = Bias.objects.filter(source_url__in=filter_values)
                                 
                        except requests.exceptions.RequestException as e:
                                response_data = {'error': str(e)}
                else:
                        form = NewsForm()


                return render(request, 'news/news_api.html', {'form': form, 'response_data': response_data, 'queryset': queryset})

Actually - I ended up declaring these variables at the top of the view and that seems to have worked…but is that the right way to do it?

def news_view(request):          
        search_term = None 
        response_data = None
        queryset = None
        if request.method == 'GET':
                form = NewsForm(request.GET)
                if form.is_valid():  
                                                    
                        search_term = form.cleaned_data['topic']
                        api_key = settings.NEWS_API
                        endpoint = 'https://api.thenewsapi.com/v1/news/all?api_token={api_key}&language=en&categories=politics&search={search_term}&sort=published_at&limit=5'
                        url = endpoint.format(api_key=api_key, search_term=search_term) 

                        try:
                                response = requests.get(url)
                                response.raise_for_status()     
                                response_data = response.json()
                                filter_values = [data['source'] for data in response_data['data']]
                                queryset = Bias.objects.filter(source_url__in=filter_values)
                                 
                        except requests.exceptions.RequestException as e:
                                response_data = {'error': str(e)}
                else:
                        form = NewsForm()


                return render(request, 'news/news_api.html', {'form': form, 'response_data': response_data, 'queryset': queryset})