Possible Django (MySQL) issue saving text with quotes

Hi. I’m trying to track down the apparent failure to save the following text in a model TextField, where the numerous quotes cause the issue (can also be caused by apostrophes). I have confirmed that the server receives the text via a JQuery Ajax call, where the parameter can be retrieved in Python using request.POST.get(“myData[content]”). I’ve used a MySQL tester to confirm it has no trouble storing the text in a table.

"aaaa aa aaa!" (aa, aaaaaaaaa, aa aa aaa aaaaaaa aaaaa’a aa aaaaaaa aaaaaaaa aaaaa aaa aaaaaaaaaa!), aaa aaaa aaa aaaa aaaaa aaaaaaa aaa aaaaaaaaa "aaaaaa aaaa" aaaaa aaaaaaaa (aaa aa aaaaaaa aaaaa), aaaaa aa aa aaaa aa aaa a/aa "aaaaaaaa a" aaaaaa, aa aa aaaaaaaaa aaa aa aaaaaaa aaa aaaaaaa aaaaaa aaaaa?

Please can anyone try saving this using a Django model, with MySQL as the database backend? Here’s the text as a string with the quotes escaped:

"\"aaaa aa aaa!\" (aa, aaaaaaaaa, aa aa aaa aaaaaaa aaaaa'a aa aaaaaaa aaaaaaaa aaaaa aaa aaaaaaaaaa!), aaa aaaa aaa aaaa aaaaa aaaaaaa aaa aaaaaaaaa \"aaaaaa aaaa\" aaaaa aaaaaaaa (aaa aa aaaaaaa aaaaa), aaaaa aa aa aaaa aa aaa a/aa \"aaaaaaaa a\" aaaaaa, aa aa aaaaaaaaa aaa aa aaaaaaa aaa aaaaaaa aaaaaa aaaaa?"

Also, please can anyone tell me whether the following filter is good (I’m new to Django / Python), where voter is a foreign key of profile, and date_created is a DateTimeField of both the Vote and Thread models (thread is a specific Thread entry):

Vote.objects.filter(voter=profile, date_created__gte=thread.date_created)

You can verify this yourself by using the Django shell (or shell_plus). Open up the shell, create an instance of your model with that text as the appropriate field value and try to save it. If there’s any error, it should be reported to you.

Hi Ken. I’m trying to diagnose the issue without access to the actual server, but I do have access to the Python source code, which I’ve been through with a fine tooth comb (I’ve likely proved the server at least receives the text, by passing it to another handler that receives the same parameters, replacing the content parameter with the troublesome text, on the client side). The server code uses a signal send_robust method to complete the transaction (with call to model save method), with content passed as a property of data structure, but I wouldn’t of thought that mechanism is the cause of the issue.

In situations like this, I never rule out anything.
If you’re going to diagnose this, you’re going to need to have this running in a “development”/“testing” environment - running locally on your own system if necessary.

1 Like

Well, I hear you, but I’m hoping someone will be willing to try a quick test to see if the text causes an issue. It could be the case that this is previously resolved Django or MySQL bug, but I’m hoping I might be lucky, and someone is able to recreate it. I think it’s reasonable to assume the signal mechanism isn’t the issue, assuming the data structure is passed by reference, i.e. the data structure isn’t serialized.

Personally, I’d find it highly unlikely that such a fundamental operation wouldn’t work. Assuming a problem with Django is always at the bottom of my “assumptions list”. You’d have to believe that something so basic would be caught by now.

Even if you don’t run the full system, you could create your own minimal model to test this just to spot-check your assumption.

As far as you know, does my filter look perfectly fine (see end of OP), in particular, comparing a models.DateTimeField (Vote.date_created) with the previously retrieved thread object, DateTimeField field value?

On how “a fundamental operation wouldn’t work”, yeah, it’s irksome, where as a matter of interest, here’s essentially what the code is doing related to the text field (where updating, which has same issue):

data = { 'id': request.POST.get('myData[id]'), 'content': request.POST.get('myData[content]') }

signal_update_data.send_robust(sender=1, data=data)

@receiver(signal_update_data)
def on_update_data(sender, data, **kwargs):
    
    myData = MyData.objects.get(id=data['id'])
    try:
         myData.content = data['content']
         myData.save()
    except Exception as e:
   		logger.warn(e)

So, I would hope you would agree that there’s nothing in the Python code that can be causing this (if only I could get a peak at the logger file, as it might be very revealing). By using a “signal”, has the site taken care of synchronizing database updates, which I assume ensures only one thread can update the database at a time?

No, I don’t agree with that.

In the absence of seeing exactly what data is being provided through the post, or a minimal test case showing a failure, I’m not able to agree.

The signal has nothing to do with that. (Nor do I see the value of using a signal in this situation - a different topic, but both of which leads me to believe you’ve got the impression that signals are doing something they don’t do.)
One common misconception is that signals are asynchronous - that they are somehow executed outside the normal flow of operations. That is not correct. Signals are dispatched to “in line” based upon the event triggering them. (See Django Anti-Patterns: Signals | Lincoln Loop among others…)

Concurrent updates are fine, provided they’re not of the same object. But you’re not locking the row so since your sequence of operations is not atomic, there is a potential race condition here.

Assuming your view has access to some “Profile” object by name “profile” I don’t see anything wrong with this. Is it not working for you?

I’ll do some further reading on “signals”, where it sounds like the original author of the code has misunderstood their purpose (my background is working with Windows servers etc, including multi-threaded code (ISAPI / C# / aspx) that utilized various database connections, such as MS SQL Server), and I note though that article is possibly incorrect when it says “With signals, we have to spread the logic across three different files.”, as this signal code is in same source file.

Thanks, on the filter. I’ve just been getting to grips with Python and Django, so I just needed someone to confirm that my filter was correct before I present the code to site owner (I’ve re-written a core function that should significantly improve performance by doing a single model query, instead of a query for every single entry in the main table, which can run into the thousands of entries!).

Or more accurately, perhaps out of date relative to more recent development. (See the section on Where should this code live in the referenced page.) While I can’t find a specific reference, something makes me think that that was the proper location for signals back around that timeframe.

However, I find the principles behind the article to still be true. I believe people tend to overuse signals in places where they’re not needed. (I also have the same opinion about Generic Foreign Keys, but that’s a different issue…)

This actually isn’t funny, given how much grief this has caused me; I was going to email someone the sample text that causes the issue and the quotes have been replaced with left and right double quotation marks, which is the workaround I implemented, but what the hell? If I edit the comment it’s showing quotes in the edit text, but the replacement characters in the displayed comment. Is there a way to actually show quotes on this forum?

Are you talking about “smart” left- and right- quotes that Windows likes to inject into documents? (Which I am seeing now in your original post - I did not catch that at first.)

If so, then that’s a case where you need to ensure that the database (and possibly your database adapter) is configured to allow unicode strings and not just Latin-1 / 7-bit ASCII / whatever you want to call the original 8-bit character set.
Django works with unicode just fine (well, about as well as anything can deal with it), but if your database isn’t configured correctly, that can cause problems.

No, I’m saying the opposite; quote characters U+0022 are the problem (not always), where if I replace them with matching U+201C and U+201D characters, the comment saves just fine. So, the forum switching the quotes for these characters is especially not helpful here, and as I say, is there a way to get the forum to actually use quotes?

Got it.

Generally speaking here, if you want to ensure text shows up literally, enclose a single line (or portion of a line) between backticks - `
This is a test "testing", 'testing'
If you have multiple lines, surround the block of text with lines of three backticks - ```.

The line before this is ```text (prevents keyword highlighting)
"testing", 'testing'
The line after this is ```

Thanks, and as you say, using the "Preformatted text" feature solved that one (if you want to use actual quotes in your regular text, I guess you’re out of luck (without font change), but I appreciate for most this is an enhancement, of showing the actual double quotes, which normally are a pain to add).

It turned out this issue was caused by us using an old version of jQuery, where one can determine the version being used from jQuery().jquery