request.POST isn't populated from an application/json POST request

Perhaps surprisingly, request.POST isn’t populated from an HTTP POST request with the form data in the request body in JSON format, even when the request’s Content-Type is application/json. Is there some reason why this is not addressed? I looked at this source in django/http/request.py:

    def _load_post_and_files(self):
        """Populate self._post and self._files if the content-type is a form type"""
        if self.method != "POST":
            self._post, self._files = (
                QueryDict(encoding=self._encoding),
                MultiValueDict(),
            )
            return
        if self._read_started and not hasattr(self, "_body"):
            self._mark_post_parse_error()
            return

        if self.content_type == "multipart/form-data":
            if hasattr(self, "_body"):
                # Use already read data
                data = BytesIO(self._body)
            else:
                data = self
            try:
                self._post, self._files = self.parse_file_upload(self.META, data)
            except (MultiPartParserError, TooManyFilesSent):
                # An error occurred while parsing POST data. Since when
                # formatting the error the request handler might access
                # self.POST, set self._post and self._file to prevent
                # attempts to parse POST data again.
                self._mark_post_parse_error()
                raise
        elif self.content_type == "application/x-www-form-urlencoded":
            # According to RFC 1866, the "application/x-www-form-urlencoded"
            # content type does not have a charset and should be always treated
            # as UTF-8.
            if self._encoding is not None and self._encoding.lower() != "utf-8":
                raise BadRequest(
                    "HTTP requests with the 'application/x-www-form-urlencoded' "
                    "content type must be UTF-8 encoded."
                )
            self._post = QueryDict(self.body, encoding="utf-8")
            self._files = MultiValueDict()
        else:
            self._post, self._files = (
                QueryDict(encoding=self._encoding),
                MultiValueDict(),
            )

The code checks for multipart/form-data and application/x-www-form-urlencoded, but not for application/json, where it essentially returns an empty data set.

This omission is a problem because although one can certainly read the data from request.body in one’s own view code, there are a lot of third-party libraries which e.g. populate forms from request.POST, and one can’t very well change the code in all those places. In the context of moving over an existing application to Django where some form submissions are done client-side by Javascript code using application/json, it would be very useful to have this functionality in core Django. I’m happy to provide a PR for this, but before doing that I’d like to know if there’s some objection by committers to doing this.

This is by design, and long-standing behaviour.

There is a (slowly advancing) DEP proposal to add content type parsing via a new .data property on the request.

That would be the way forward.

Thanks for the link. Interesting, but it doesn’t address the fact that many libraries use request.POST to get the POST data, and who knows when they’ll all move over to use the data attribute? A little disappointing for my current use case, but hey ho. Can you (or anyone) shed light on why application/json was left out?

I don’t know if it’s the reason, but a reason is that it would break everything that relies upon request.POST being HTML form-submission data.

Django was built at a time before JSON-based APIs were a “thing”, and that the “X” in “AJAX” really did refer to “XML” as being the web-service payload-of-choice. (The JSON format itself was not standardized until about 2014. In fact, my first real Django project involved passing XML around in request.body.)

Side note: My reading of DEP 0015 is that it adds these new features without removing request.POST, so this will not affect existing libraries.

1 Like

How exactly would it do that? Asking because I’m trying to understand. If a JSON object is POSTed which corresponds to a dict, then how is it different if that JSON dict populates request.POST, the same way as the other currently handled mime types do?

Yes, but they have been a thing for a long while now.

I think you might be misunderstanding my reference to existing libraries. It’s not that they’ll break because of this DEP, but that they won’t be useful to their users for contexts where JSON data is POSTed until they get around to using the functionality proposed by the DEP.

I’m facing this right now in a situation where a legacy application is POSTing JSON, but a third party Django library can’t see that data because it’s looking in request.POST. That library defines a lot of forms, all of which look for POST data in request.POST, so it doesn’t seem feasible to work around it. Nor is this the only third-party library which looks in request.POST.

What is submitted in the POST data portion of the request is not a dict, nor is it JSON. What you are looking at as request.POST is the result of Django reformatting the submitted data and creating a Dict-like object from what was submitted. They’re two different things.

By definition and documentation, request.POST creates a QueryDict from form data. All other uses require the use of request.body.

There are constraints and limitations of what can be submitted from both formats, and there’s not a strict 1-1 correspondence between the two.

For more information about what is actually submitted, see:

I would find that surprising. But in the absence of any details, it’s all conjecture.

Hi @vsajip, there is no good reason why request.POST couldn’t also parse JSON. That said, adding it at this point would maybe be backwards incompatible. The code you initially linked for instance does not touch request.body if the mimetype doesn’t match. So existing code that checks for request.POST and then falls back to reading request.body might be broken if we all of a sudden started to also parse other mimetypes here.

Since we want to move away from upper case attribute names anyways we thought it would be a good idea to do that in one go with better content type handling where we would also have the chance to introduce new semantics etc… (Also having the new parsing only on the new attributes might motivate users to switch earlier ;))

But yeah, I would say part is history and part our reluctance to introduce backwards incompatible changes to such an essential part of the system.

Cheers,
Florian

What details would you want? The library I’m talking about is django-allauth. Anyway, I think we’re talking at cross-purposes; I’m quite aware of what request.POST is, and how it’s currently populated, thanks.

Yes, I understand, but I wasn’t envisaging changing request.body, just populating request.POST from it. As I see it, code looking for POST data would normally either check request.POST and use it, or check content_type and, if application/json, get the data from request.body and go on its way. The breaking code would have to check for a non-empty request.POST and a content_type of application/json and then blow up because those are (currently) inconsistent. I’m not saying there’s no code out there that does that, but it sounds a little far from the happy path!

Well it would blow up because I think request.body might be empty since it is already read then by request.POST (but I might be miss-remembering). Agreed that this is far away from the happy path, but if we add JSON here now we would have to answer the question of why we don’t allow yaml as well etc… Hence the new DEP to add custom parsers basically (or at least the foundation) and move away from request.POST completely (because you could also send application/json with a PUT which wouldn’t get parsed either no matter what we do for request.POST).

I think perhaps you are misremembering - request.body is not a stream, but a bytestring. It remains unaffected after a json.loads(self.body.decode('utf-8')) operation - I verified this using a strategically placed breakpoint.

Are you sure that’s not a straw man? I don’t know of people POSTing/PUTting YAML, though doing that with JSON is ubiquitous.

That sounds fine as a strategy, but of course people using third-party libraries won’t be able to take advantage until those libraries move over to use the new functionality - and if this is seen as a big change, would it go in before Django 6.0, even assuming an implementation were ready to go?

Ah yes, you’d only run into issues if you manually try to .read() on the request – I knew why I said I might be miss-remembering :smiley: Sorry for not looking it up first!

Agreed on that, though I am pretty sure it will pop up. And even if it is not about YAML per se the question would be why we’d hardcode JSON and not make it plug able. Which is a fair question for a framework like Django. For example I do a bit with FHIR which uses application/fhir+json as mime-type. Granted it does optionally support application/json as well but if the client sends application/fhir+json you have to handle that – so you still would want some hooks as opposed to simply hardcoding application/json.

Yes, given the current status of 5.2 this would most likely be in 6.0 if at all. Yes, Django is slow moving nowadays (for better or worse).

I’m not against the approach talked about in the DEP. The main reason for doing something now would be to support the use case I outlined earlier. There’s no reason other than fear of compatibility issues for not adding support for JSON in the short term, and supporting a pluggable system going forward. Why hardcode JSON? Because it’s ubiquitous, just like the other two currently hardcoded mime types.

I know this discussion is academic - due to the worry about compatibility, Django maintainers will not allow changes in this area. Hopefully the thread will provide information to others who run into this issue. Thanks, @apollo13 and @KenWhitesell, for the responses.