how to read bytea in django ?

Using Django
1

hi, so i am fetching data from postgresql but image doe not show in html web page.

My code:

def home(request,colors):

``C = Retailers.objects.filter(colors=colors)``
``context = {``
`	`"C":C`` `` ``}``
``return render(request, "Prods/Home.html",context)``

{% for c in C %}

<div class="pa2 mb3 striped--near-white">

<div class="pl2">

<a href = {{ c.links }}> <p class="mb2">{{ c.title }} </a>

</p>

<p class="mb2">Price: {{ c.price }}</p>

<img src = "data:image/jpeg;base64,{{ c.images }}" >

</div>

</div>

{% endfor %}

Everything else it shows but not the image, how can i make it read it

2

What is the field definition for the images field in Retailers?

Unless you’re storing it as base64 data, you will need to convert it to that before sending it.

3

hi first it was models.Textfield() (But that was my bad so i changed it to)

models.BinaryField()

but when i insert images, i get error cannot convert str to bytes :frowning:

It works well on different python script.

Q = "select title,images from retailers"

cur.execute(Q)

Query = cur.fetchall()


for r in Query:


    image_data = io.BytesIO(r[2])

    file_bytes = np.asarray(bytearray(image_data.read()), dtype=np.uint8)

    img = cv2.imdecode(file_bytes, cv2.IMREAD_COLOR)
    
 
    cv2.imshow(r[1],img)
    
    cv2.waitKey(0)

    cv2.destroyAllWindows()
4

It’s that error message (the complete traceback) that would be of value to see. (My impression from this statement is that you’re getting the error when you’re trying to add the image to the field - a slightly different issue.)

How are you populating these models? Is this pre-existing data that you are just accessing from Django, or is this all built in Django?

(I’m having a hard time seeing how you would get an error about converting str to bytes when reading this data if the data is already stored as binary data in the database. You mentioned that you changed this from a TextField to a BinaryField. Was the underlying table also changed?)

I understand what you’re saying here, and not intending to be blunt or rude about it, but that’s not particularly relevant. That example isn’t Django-based code working with the ORM and trying to generate a base64 encoding of an image. There’s not really any useful information to be gathered from it.

5

hi sorry for not clarifying before, so this is how i insert data into my models fields.

My models File:

from django.db import models

# Create your models here.


class Retailers(models.Model):

	links = models.TextField(unique=True)

	country = models.TextField()

	retailer = models.TextField()

	category = models.TextField()

	title = models.CharField(max_length=300)

	price = models.DecimalField(max_digits=10, decimal_places=2)

	images = models.BinaryField()

	colors = models.TextField()



	def __str__(self):
		
		return self.title

Then my code to insert data into the respective fields: (in manage.py Shell)

import pandas as pd

from Prods.models import Retailers





R = pd.read_json("R1.json")


links = R["links"].tolist()
country = R["country"].tolist()
retailer = R["category"].tolist()
title = R["title"].tolist()
price = R["price"].tolist()
images = R["images"].tolist()
colors = R["colors"].tolist()



for L,CT,R,T,P,I,CL in zip(links,country,retailer,title,price,images,colors):

 K = Retailers(links = L ,country = CT ,retailer = R, title = T, price = P ,images = I,colors = CL)

K.save()

i get a long error and in the end it says

TypeError: can’t escape str to binary

6

Now, in the general case, I would normally point out that it’s frequently important and useful to supply the complete traceback. There are many situations where the underlying cause is not the last message displayed, but something higher up in the stack.

Fortunately, this is not one of those situations.

I’m not familiar enough with pandas to know how it interprets / translates the incoming JSON to those individual fields, but it is apparent that it is storing it in images as a string. What’s not clear is exactly how that string is encoded - and that may make a difference here.

Do you have a small sample of what an instance of images looks like immediately after this line?:
images = R["images"].tolist()

If not, can you post a small section of the JSON being handled? (Specifically interested in the images key.)

7

hi really sorry for the late reply.

Here is the sample of the images string.

I even tried the method below to store the images, but it says filename too long

8

That format is actually text. It’s a text representation of binary data.

Storing it as a text field could be appropriate - it really depends upon what all you’re going to be doing with it.

If you’re only ever going to be displaying them as a data URL within an img tag, you could directly store these as base64.

Unfortunately, having the ‘\x’ as the first two characters makes this trickier than I think it should be. Python interprets ‘\x89504e470d0a’ as “\x89” “504e470d0a”, where “\x89” is considered 1 character.

For example, from the regular Python shell:

>>> s = '\x89504e470d0a'
>>> s
'\x89504e470d0a'
>>> print(s)
504e470d0a
>>> s[0]
'\x89'
>>> s[1:]
'504e470d0a'

Getting rid of that notation and coming up with something usable requires a little bit of trickery.

There are other ways, but what I’ve come up with is:

>>> new_string = format(ord(s[0]), "x") + s[1:]
>>> print(new_string)
89504e470d0a
>>> new_string
'89504e470d0a'

Once you have the string in a usable format, you can now convert it to bytes and then to a base64 encoding.

>>> binary_data = bytes.fromhex(new_string)
>>> binary_data
b'\x89PNG\r\n'

Finally, you can turn it into base64:

>>> import base64
>>> base64_bytes = base64.b64encode(binary_data)
>>> base64_bytes
b'iVBORw0K'
>>> base64_string = base64_bytes.decode()
>>> base64_string
'iVBORw0K'

It’s that last value (base64_string) you’ll want to use in your template for display.

What you choose to store in the dabase is up to you.

  • You can store the original string, as text, and do these transformations in the view.
  • You can do the conversion to bytes and store it as binary data.
  • You can finish the conversion and store the base64 string.
9

thanks you so much, is there a way to you know do it automatically. (as soon as i insert the images from my pandas list to Django models) ?

for examply lets say i have a title field and when ever i save it, it automatically makes all the letters UPPER CASE.

same with how can i convert the string into base64 automatically ?

Thanks for your time :slight_smile:

10

Yes, add the appropriate statements to your code when you’re reading / pre-processing the data.

What I provided you were the functions you may find most useful with doing this. You need to adapt this to your processes.

11

thanks you so much :slight_smile: you are awesome, have a great day

12

hi i have a problem, so i lopped over and saved the images in TextField model.

but python is reading it as single string so it does not have 0 or 1 index.

what should i do here ?

should i start from the binary_data part of your code. Thanks :slight_smile:

13

Yes it does. Strings are iterables and can be referenced by index.
See 3. An Informal Introduction to Python — Python 3.12.0 documentation and the various examples within that section.

14

hi can you help me i get error, while doing the following code in my views file

def home(request,colors):

	C = Retailers.objects.filter(Color=colors)


	for b in C:
		A = b.Images.replace("\\x89","")

		binary_data = bytes.fromhex(A)

		base64_bytes = base64.b64encode(binary_data)

		base64_string = base64_bytes.decode()




	context = {

		"C":C,

		"B": base64_string
	}

	return render(request, "Prods/Home.html",context)

Error:

Internal Server Error: /favicon.ico/
Traceback (most recent call last):
  File "/home/firaki/.local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/firaki/.local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/firaki/Desktop/images/blog/Prods/views.py", line 30, in home
    "B": base64_string

UnboundLocalError: local variable 'base64_string' referenced before assignment

Sorry for bothering you. :slight_smile:

15

One cause for this error would be a case where your query doesn’t return any data.
If C is an empty queryset, then the for loop doesn’t execute and the variable base64_string never has a chance of being initialized.

16

HI, thanks a lot, you were right. was not calling it properly and i found out that i had to remove the \x part to show the image, and now it shows the image perfectly :slight_smile:

But now one issue i have is saving and converting the text to base_64 automatically as soon as i insert the text into the text field. I tried the following below, it returned no errors but it does not remove the \x part

Maybe i am not saving it properly

class Retailers(models.Model):

	Links = models.CharField(max_length=500,unique=True)

	Title = models.CharField(max_length=300)

	Images = models.TextField()

	Color = models.CharField(max_length=40)



	def __str__(self):
		
		return self.Title



	def Binary_data(self):

		A = self.Images.replace("\\x","")

		binary_data = bytes.fromhex(A)

		base64_bytes = base64.b64encode(binary_data)

		base64_string = base64_bytes.decode()

		self.Images = base64_string

		return self.Images

Sorry for bothering you again :slight_smile:

EDIT I FIGURED IT OUT:

def save(self, *args, **kwargs):

	A = self.Images.replace("\\x","")

	binary_data = bytes.fromhex(A)

	base64_bytes = base64.b64encode(binary_data)

	base64_string = base64_bytes.decode()

	self.Images = base64_string

	super().save(*args, **kwargs)