SynchronousOnlyOperation error in production (Heroku) but not local development

Background:
I am training a stable_baselines3 model using pytorch in a celery task. When doing this locally all works as expected but once deployed to heroku I receive the following error:

app[worker.1]: Traceback (most recent call last):
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
app[worker.1]: R = retval = fun(*args, **kwargs)
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 650, in __protected_call__
app[worker.1]: return self.run(*args, **kwargs)
app[worker.1]: File "/app/manekineko/agents/tasks.py", line 96, in train_model
app[worker.1]: model.model.learn(total_timesteps=model.timesteps)
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/stable_baselines3/ppo/ppo.py", line 307, in learn
app[worker.1]: total_timesteps, callback = self._setup_learn(total_timesteps, eval_env, callback, eval_freq,
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/stable_baselines3/common/base_class.py", line 536, in _setup_learn
app[worker.1]: self._last_obs = self.env.reset()
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 61, in reset
app[worker.1]: obs = self.envs[env_idx].reset()
app[worker.1]: File "/app/.heroku/src/gym-stock-trading/gym_stock_trading/envs/stock_trading_env.py", line 527, in reset
app[worker.1]: self._initialize_data()
app[worker.1]: File "/app/.heroku/src/gym-stock-trading/gym_stock_trading/envs/stock_trading_env.py", line 334, in _initialize_data
app[worker.1]: self.asset_data, self.previous_close = next(self.market_data)
app[worker.1]: File "/app/manekineko/agents/tasks.py", line 61, in _yield_market_data
app[worker.1]: for stock_data in stock_data_queyset:
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/db/models/query.py", line 276, in __iter__
app[worker.1]: self._fetch_all()
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
app[worker.1]: self._result_cache = list(self._iterable_class(self))
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
app[worker.1]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
app[worker.1]: cursor = self.connection.cursor()
app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
app[worker.1]: raise SynchronousOnlyOperation(message)
app[worker.1]: django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

Essentially the function queries the database and uses a generator to feed it to the model as needed. I’ve read through the docs and am still struggling to understand the fundamentals of what is causing the problem, especially if it works locally.

I appreciate any help, especially with an explanation of the role async plays here.

function I believe is causing the error

def _yield_market_data(model, mode):
    while True:
        if mode == 'train':
            stock_data_queyset = model.agent.asset.stockdata_set.filter(
                data_type='aggregates',
                timespan='minute',
                market_hours=True
            ).filter(
                start_time__lte=datetime.date.today()\
                    - timedelta(model.evaluation_days + 1),
                start_time__gte=datetime.date.today()\
                    - timedelta(model.evaluation_days + model.training_days)
            ).order_by(
                'start_time'
            )
        elif mode == 'evaluate':
            stock_data_queyset = model.agent.asset.stockdata_set.filter(
                data_type='aggregates',
                timespan='minute',
                market_hours=True
            ).filter(
                start_time__gte=datetime.date.today()\
                    - timedelta(model.evaluation_days),
            ).order_by(
                'start_time'
            )

        for stock_data in stock_data_queyset:
            asset_data = pd.read_json(stock_data.json_data)
            previous_close = stock_data.previous_close

            yield asset_data, previous_close

In case anyone come across this. I ended up just placing the function call in a thread and it resolved the issue. Still not sure I understand why this is necessary or why it works on local and not production but its working now.

1 Like

maybe it related to process and subprocess ,

I know this is pretty old, but I was working on a django 3.2.12 project on Heroku and ran into the same very frustrating error. For me the issue was clearly the call to the database on Heroku. Does anybody have some further insight on this? (please ignore that having a simple unfiltered query might be bad practice)

@duhfrazee threading was a good hint, this is what worked for me in the end:

import threading

import pandas as pd
from django.db import models

class LoadDBDataThread(threading.Thread):
    def __init__(self, model: models.Model):
        super().__init__()
        self.model = model
        self.result = None

    def run(self):
        """Override the run method to execute the desired function."""
        self.result = self.load_from_DB(self.model)

    def load_from_DB(self, model: models.Model):
        """Load table from DB as pandas Dataframe."""
        return pd.DataFrame(list(model.objects.all().values()))
    

def load_DB_data(model: models.Model):
    """Load table from DB as pandas Dataframe."""
    # Create an instance of the custom thread class
    thread = LoadDBDataThread(model)

    # Start the thread
    thread.start()

    # Wait for the thread to finish
    thread.join()

    # Return the result from the thread instance  
    return thread.result