Migrating from WP RESTful API to PostgreSQL. What am I doing wrong here?

Traceback (most recent call last):
  File "~/project/packages/server/manage.py", line 27, in <module>
    main()
  File "~/project/packages/server/manage.py", line 23, in main
    execute_from_command_line(sys.argv)
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/project/packages/server/api/management/commands/populate_venues_metadata.py", line 18, in handle
    ven    = Venue.objects.get(old_wp_id=ven_id)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 645, in get
    num = len(clone)
          ^^^^^^^^^^
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 382, in __len__
    self._fetch_all()
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 123, in __iter__
    for row in compiler.results_iter(results):
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1500, in apply_converters
    value = converter(value, expression, connection)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/project/packages/server/.venv/lib/python3.12/site-packages/django/db/models/fields/json.py", line 94, in from_db_value
    return json.loads(value, cls=self.decoder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/py/cpython@3.12.3/lib/python3.12/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

What I am doing is using Python Command to make a request to my current WP RESTful API database to retrieve data from a specified endpoint and saving it to a PostgreSQL database. I’m able to get a response from my WP database but no matter what I do, I always get this error: TypeError: the JSON object must be str, bytes or bytearray, not dict. I have been scouring Google for the last 3 days and cannot figure out why this is continuing to happen.

I’ve already run migrations and migrate and everything is flowing as expected.

The models I am using are as follows:

class Venue(models.Model):
    id = models.BigIntegerField(primary_key=True)
    name = models.CharField()
    slug = models.CharField()
    description = models.TextField(blank=True, null=True)
    image = models.CharField(blank=True, null=True)
    capacity = models.IntegerField(blank=True, null=True)
    address = models.CharField()
    geo = models.JSONField()  # This field type is a guess.
    wr_id = models.CharField()
    old_wp_id = models.BigIntegerField()
    parent = models.BigIntegerField()

    class Meta:
        managed  = True
        db_table = 'rd_venues'

    def __str__(self) -> str:
        return self.name

    def __getitem__(self, key: str):
        return self[ key ]

class VenueMeta(models.Model):
    id = models.BigIntegerField(primary_key=True)
    venue = models.ForeignKey(Venue, models.DO_NOTHING)
    street = models.CharField(blank=True, null=True)
    locality = models.CharField(blank=True, null=True)
    region = models.CharField(blank=True, null=True)
    postal_code = models.CharField(blank=True, null=True)
    country = models.TextField()  # This field type is a guess.
    timezone = models.CharField()
    union_id = models.BigIntegerField()
    place_id = models.CharField(blank=True, null=True)

    class Meta:
        managed = True
        db_table = 'rd_venues_metadata'

    def __str__(self) -> str:
        return self.venue.name

    def __getitem__(self, key: str):
        return self[ key ]

My request is as follows:

def venues():
    return requests.get( 'https://my.custom.wp/endpoint/venues/' ).json()

My python command is as follows:

from typing import Any

from django.core.management.base import BaseCommand

from api.models import Venue, VenueMeta
from api.requests import venues


class Command(BaseCommand):
    def handle(self, *args: Any, **options: Any) -> str | None:
        data = venues()

        for vm in data['data']:
            ven_id = vm['id']
            ven    = Venue.objects.get(old_wp_id=ven_id)

            if ven:
                print( 'Venue ' + ven['name'] + ' exits' )
                continue
            elif VenueMeta.DoesNotExist:
                entry = VenueMeta.objects.create(
                    venue       = ven.first(),
                    street      = vm['schemaOrg']['addressStreet'],
                    locality    = vm['schemaOrg']['addressLocality'],
                    region      = vm['schemaOrg']['addressRegion'],
                    postal_code = vm['schemaOrg']['postalCode'],
                    country     = vm['schemaOrg']['addressCountry'],
                    timezone    = vm['timezone'],
                    union       = ( 1 if vm['is_home'] == True else 0 ),
                    place_id    = vm['place_id'],
                )
                entry.save()
                print('Venue ' + vm['name'] + ' metadata imported')
            else:
                continue

        return super().handle(*args, **options)

Any insight would be much appreciated. Cheers :beers:

Any chance the issue can be from this field?

It seems you’re kind of lost on general debugging, so here are some suggestions:

  • Print/log out the ven_id on each iteration.
  • Rerun the command
  • Find the ven_id that was running when the code error from the logs
  • Open up the shell
  • Run Venue.objects.get(old_wp_id=id from your log)
  • That might give you minimal steps to reproduce the issue.

If it does, try

Venue.objects.defer(“geo”).get(old_wp_id=ven_id)

If that works, but the original get doesn’t, we need to investigate the data that’s in the geo column in your database.

1 Like

I think the __getitem__ method is the problem…
Remove the method and proceed with migration.

Do not use a __getter__, but use the .__dict__ command to convert the object to a dict.

If that doesn’t work, make the contents of ~/project/packages/server/api/management/commands/populate_venues_metadata.py public.

1 Like

@massover @white-seolpyo I took parts from each of your suggestions and got things to work!

@massover Inside the Venue model, I changed JSONField to TextField.

@white-seolpyo I removed __getitem__ as suggested.

THANK YOU BOTH SO MUCH!!! Cheers :beers: