Saving API data and call it in views

Hi there,

I have written the following: basic idea would be for the macro_char() in views.py to consumer an API, save the data to a model called Macro and from the model spit out the data within the context. Getting the following error message when the template is rendered, "local variable ‘data’ referenced before assignment

Would be great if folks could help out, please, cheers. Ismail

def macro_chart(request):
   try:
   	url = 'https://api.db.nomics.world/v22/series/BOE/3687/IUAAAJNB?observations=1'
   	response = requests.get(url)
   	response_json = json.loads(response.content)
   	time.sleep(0.5)
   	data_attr = {
   		'dataset_name': response_json['series']['docs'][0]['dataset_name'],
   		'periods': response_json['series']['docs'][0]['period'],
   		'dataset': response_json['series']['docs'][0]['value']
   		}
   	data = Macro.objects.create(**data_attr)

   	data.save()
   		
   except Exception as e:
   	response = "Opps something is wrong with the API"

   context = {
   		"periods": data.periods, 
   		"dataset": data.dataset
   	}
   return render(request, "macro_main.html", context)

Just a note for future reference - when you post code here, please enclose it between lines consisting of only three backtick - ` characters. That means you’ll have one line of ```, then your code, then another line of ```. This preserves the formatting of the code making it a lot easier to read - especially with statements where indentation is critical.

example:

# The line above this is ```
def function(self):
    return self
# The line after this is ```

If you can, please edit your post to put the ``` before and after the code.

Ken

1 Like

Cool, ok, so the problem is that the api call failed for whatever reason. This means that your code didn’t execute the data=... line, leaving the variable data undefined.

After the exception is processed, you then try to build a dict named context which is making a reference to data - but since it wasn’t assigned, the error is being thrown.

I assumed because the fields in the Model match the fields in data_attr=, then the syntax would work. In models.py I have the following code

from django.db import models
from datetime import datetime, date

# Create your models here.
class Macro(models.Model):
	dataset_name = models.CharField(max_length=30, blank=True)
	periods = models.DateField(auto_now_add=False, auto_now=False, blank=True)
	dataset = models.DecimalField(max_digits=5, decimal_places=2)

	def __str__(self):
		return self.dataset_name

The failure could be anywhere within the try block. You’re swallowing the exception and not providing yourself with any debugging information. You’ll get the same symptoms regardless of the actual problem.

Thanks, Kevin, that really helpful. The issue seems to be the data I’m receiving from the API is time-series data, after parsing the code below, I get list. For example periods would be [‘1976’, ‘1977’, ‘1978’, ‘1979’] and dataset would be [‘5.2’, ‘4.7’, ‘5.6’, ‘1.3’]

data_attr = {
		'periods': response_json['series']['docs'][0]['period'],
		'dataset': response_json['series']['docs'][0]['value']
		}

My model has the following fields, and I suspect it’s not the correct one:

class Macro(models.Model):
	dataset_name = models.CharField(max_length=30, blank=True)
	periods = models.IntegerField(blank=True, null=True)
	dataset = models.DecimalField(max_digits=5, decimal_places=2)

	def __str__(self):
		return self.dataset_name

What is the best way to store such data in a model, for example, ArrayField, or potentially convert the data to Pandas Dataframe and store in the model?

This isn’t a case of a choice being “correct” or “incorrect”, it’s extremely context sensitive. I’d say it’s really going to depend upon what you’re going to do with it after you’ve gotten it stored.

In this case, I’d be considering how much data I’m going to have and what queries are going to be run on it. For example, am I just going to be retrieving values by year? Or am I going to want to do something like “Find all years where the value is > 5”? Am I going to have 10 sets of data like this? Or is it going to be more like 10,000? Or even 10,000,000? (Or more?)

If I don’t have those answers, then I tend to start with a fully-normalized structure, which would be something like (shortened, and written off-the-cuff):

class DataSet(Model):
    dataset_name = models.CharField(max_length=30, blank=True)
    date_retrieved = models.DateTimeField(...)

class Data(Model):
    dataset = models.ForeignKey(DataSet, ...)
    year = models.IntegerField(...)
    value = models.DecimalField(...)

But in the small/trivial cases (< 1000ish rows) your structure isn’t going to make a whole lot of difference. You could just as easy serialize the entire response and store it in a JSONField without causing you any serious difficulties. I’ve got one table (around 3000 rows) where I’ve done that quite successfully. PostgreSQL effectively keeps the entire row in cache in memory. There’s no visible lag, latency, or delay with performing JSON-based queries on that table compared to a more normalized structure.

Ken