# Discussion on Configurable Content Type Parsing Project

**URL:** <https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865>\
**Category:** Google Season of Code\
**Created:** [February 15, 2023, 8:48pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865 "2023-02-15T20:48:43Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 15, 2023, 8:48pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/1 "2023-02-15T20:48:43Z")

</div>

Hi everyone, I want to contribute to this project as part of GSOC 2023, I’ve created this discussion to get some more details about the project and get going in the right direction!

1. I went through the last commit done by @carltongibson here: [Refs #21442 – Added content-type aware request.data property. · django/django@d1ba27d (github.com)](https://github.com/django/django/commit/d1ba27df5d9b2c1507977adb0c48b7bf32bfaf08#diff-36b4f64c7520b4980156f634f551516ba9f8d50df510ce5dffd3287ecbbb4404)  
Is the request.POST property which was widely used earlier for data extraction from a request object, completely removed?

2. Is request.POST replaced by request.data and will it serve all the purpose?

3. What exactly are the acceptable ‘content-types’ for the middleware parsers that can return a ‘True’ boolean value?

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 16, 2023, 7:29am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/2 "2023-02-16T07:29:34Z")

</div>

Hi @udbhavsomani — thanks for looking at this.

Let me respond to your points:

On 1. and 2. `request.POST` will become an alias for the new `request.data`, so that existing code will continue to work. It will be scoped for only POST requests, and only form data encoded parsing (maybe) — so that its behaviour is unchanged.

On 3. The idea is **any content type** a user wants to provide a parser for. Django will provide form data and JSON parsing, but others that might be used could be MessagePack, Protobuf, XML, yaml, … — it goes on. Also there are specialisations, like `application/vnd.github+json` and so on. All Django needs to provide is the API for a parser — likely a `can_accept()` and `parse()` 🤔. The idea would be to have a list and use the first parser that said yes to `can_accept` — allowing a specialist parser a first shot, before falling back to a more generic parser (for the +json type examples).

I want to pick up my WIP PR shortly, and aim for it to be complete before the start of the GSoC season — help there could include test cases for the (single and multipart) JSON parsing, which is next on my list. A GSoC project would be based around generalising the `Parser` bit, which I’m not aiming for in the first phase.

How’s that?

Kind Regards,

Carlton

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 16, 2023, 8:24am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/3 "2023-02-16T08:24:46Z")

</div>

Hi @carltongibson - Thank you for such a fast and informative response.  
I have the gist on what we are trying to achieve here.

I see that you have already made a few tests here ([Refs #21442 -- Added content-type aware request.data property. · django/django@d1ba27d · GitHub](https://github.com/django/django/commit/d1ba27df5d9b2c1507977adb0c48b7bf32bfaf08#diff-378adecd3fe6f72607f05dfc2a6fd2a259106e8beb502833230b61061a7412d4))

If it is okay by you, can I try writing these tests which will also help me to get a better understanding on this?

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 16, 2023, 8:34am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/4 "2023-02-16T08:34:06Z")

</div>

Super! If you’d like to join in, I’m happy to have the help. 😊

First of is straight `application/json` parsing. (So a request.data gives parsed Python object from a JSON body.)

Then it’s multipart examples with JSON parts.

With those bootstrapped we can look at more examples. If you wanted to make a PR against my PR branch (if that makes sense 🤯) that would be awesome. We can discuss details there.

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 16, 2023, 9:04pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/5 "2023-02-16T21:04:04Z")

</div>

Hey @carltongibson!  
I set up the repo locally on my machine and ran the tests.py. I tried writing a test function for JSON acceptance, but it seems like `request.data` returns an empty querydict when content-type is changed to `application/json`. The data is there in the `request.body` from where I can load and match it.  
Here is the test function that I tried.

```auto
    def test_json_data(self):
        """
        JSON from various methods.
        """
        for method in ["GET", "POST", "PUT", "DELETE"]:
            with self.subTest(method=method):
                payload = FakePayload("{\"key\": \"value\"}")
                request = WSGIRequest(
                    {
                        "REQUEST_METHOD": method,
                        "CONTENT_LENGTH": len(payload),
                        "CONTENT_TYPE": "application/json",
                        "wsgi.input": payload,
                    }
                )
                print(request.data) # <QueryDict: {}>
                self.assertEqual(json.loads(request.body), {"key": "value"}) # OK

```

Any help on what is happening and what am I missing out on?

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 17, 2023, 6:56am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/6 "2023-02-17T06:56:51Z")

</div>

Yes, that looks right, and you’ve discovered the current behaviour.

The task then is to make that pass.

If you look in `django/http/request.py` for the `HttpRequest._load_post_and_files()` method, you’ll see a branches where different content types are handled. Current behaviour there is to drop out the bottom, in the `else` branch. This is the bit that we’ll eventually make pluggable, but, for now we can add a branch for `application/json`.

Hopefully that makes sense?

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 17, 2023, 10:11pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/7 "2023-02-17T22:11:29Z")

</div>

Oh, I see the code there.  
I’ll try adding an `elif` block for handling the `application/json` content\_type to populate the `request.data` object.

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 17, 2023, 10:59pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/8 "2023-02-17T22:59:21Z")

</div>

I was actually able to parse JSON and pass the test case successfully.  
I went ahead and created a PR to your WIP branch (🙈).  
Request you to kindly review it here: [[4.2] Added JSON parsing for request.data property by udbhavsomani · Pull Request #16570 · django/django (github.com)](https://github.com/django/django/pull/16570)

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 18, 2023, 7:32am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/9 "2023-02-18T07:32:06Z")

</div>

Good work! I’ll review properly in the week.

Next step would be the same for multipart requests.

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 19, 2023, 7:10pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/10 "2023-02-19T19:10:29Z")

</div>

Thank you!  
Sure! I’ll try to make progress on the multipart requests containing JSON.

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 19, 2023, 8:06pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/11 "2023-02-19T20:06:38Z")

</div>

I had a few blockers, your help here would be highly appreciated.

1. I tried this:

```auto
def test_POST_multipart_form_json_data(self):
        payload = FakePayload(
            "\r\n".join(
                [
                    "--boundary",
                    'Content-Disposition: form-data; name="json_data"',
                    "Content-Type: application/json",
                    "",
                    "{\"key\": \"value\"}",
                    "--boundary--",
                ]
            )
        )
        request = WSGIRequest(
            {
                "REQUEST_METHOD": "POST",
                "CONTENT_TYPE": "multipart/form-data; boundary=boundary",
                "CONTENT_LENGTH": len(payload),
                "wsgi.input": payload,
            }
        )
        print(request.data) # <QueryDict: {'json_data': ['{"key": "value"}']}>
        self.assertEqual(request.data, {'json_data': ['{"key": "value"}']}) # OK

```

This test case passes with the given output. Is it correct though? As in, is this the output that should be acceptable?

1. There is this function:

```auto
def test_body_after_POST_multipart_form_data(self):
        """
        Reading body after parsing multipart/form-data is not allowed
        """
        # Because multipart is used for large amounts of data i.e. file uploads,
        # we don't want the data held in memory twice, and we don't want to
        # silence the error by setting body = '' either.
        payload = FakePayload(
            "\r\n".join(
                [
                    "--boundary",
                    'Content-Disposition: form-data; name="name"',
                    "",
                    "value",
                    "--boundary--",
                ]
            )
        )
        request = WSGIRequest(
            {
                "REQUEST_METHOD": "POST",
                "CONTENT_TYPE": "multipart/form-data; boundary=boundary",
                "CONTENT_LENGTH": len(payload),
                "wsgi.input": payload,
            }
        )
        self.assertEqual(request.POST, {"name": ["value"]})
        with self.assertRaises(RawPostDataException):
            request.body

```

Please help me understand what this test really does?

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 21, 2023, 1:54pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/12 "2023-02-21T13:54:16Z")

</div>

> [@udbhavsomani](#):
>
> `print(request.data) # <QueryDict: {'json_data': ['{"key": "value"}']}>`

It doesn’t look like the JSON is parsed here. You have a string no? `'{"key": "value"}'`.

I need to have a think about `MultiDict`’s `get()` vs `get_list()` but — I’d expect a (list of) dict(s) in the `json_data` key…

```auto
assert request.data.get("json_data") == {"key": "value"}

```

> [@udbhavsomani](#):
>
> Please help me understand what this test really does?

See where it’s raised in `body`:

```auto
    # in django/http/request.py
    def body(self):
        if not hasattr(self, "_body"):
            if self._read_started:
                raise RawPostDataException(
                    "You cannot access body after reading from request's data stream"
                )
         ...

```

Since multipart parser reads from `request._stream` (via `read()`) — this enforces (basically) that you either use the parsed `POST` method or you handle this yourself via `body` but not really both.

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 22, 2023, 7:35am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/13 "2023-02-22T07:35:12Z")

</div>

> [@carltongibson](#):
>
> You have a string no?

Oh, right, I’ll have to parse and return a json value.

> [@carltongibson](#):
>
> I’d expect a (list of) dict(s) in the `json_data` key

Got it. Will do the required changes.

> [@carltongibson](#):
>
> not really both

Ahan, I see the handling of redundant parsing here. Thank you!

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 22, 2023, 9:47pm UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/14 "2023-02-22T21:47:49Z")

</div>

Hi @carltongibson  
I was working on the multipart/form-data with json parts and I am kind of stuck at the multiparser bit.  
I was looking at the code in `django/http/multipartparser.py` for the parse function.  
Under this loop (which, imo, runs for every part of the multipart request):  
`for item_type, meta_data, field_stream in Parser(stream, self._boundary):`

I tried to add this code for json parsing, but got stuck on how to access the data from the stream:

```auto
try:
    sub_content_type = meta_data["content-type"][0]
    sub_content_type = sub_content_type.strip()
    if sub_content_type == 'application/json':
        pass # access data by using the exhaust() method maybe?
    except (KeyError, IndexError, AttributeError):
        continue

```

Is this the correct approach?

And for the gsoc configurable type parsing project, we would require a similar, new class with multiple parsers and use this multipartparser as the default one?

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 23, 2023, 9:34am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/15 "2023-02-23T09:34:44Z")

</div>

Yes, something along these lines. Once we know the part is JSON we need to get all of its data in order to parse.

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 23, 2023, 9:36am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/16 "2023-02-23T09:36:27Z")

</div>

For single part parsing we just use the list of parsers. Then there’s a multipart parser that uses the same list for each part (or so I imagine… but it’s not settled yet: part of the project is determining what’s needed.)

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 24, 2023, 8:28am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/17 "2023-02-24T08:28:19Z")

</div>

Hi @carltongibson!  
I have made changes to my PR here: [Refs #21442 – Added JSON parsing for request.data property. by udbhavsomani · Pull Request #16570 · django/django (github.com)](https://github.com/django/django/pull/16570)

It would be really kind of you to please review it. I have made the test cases for multipart+JSON type request objects as well as modified the existing parser bit which now returns (list of) dict(s) for the given JSON key.

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![carltongibson](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/carltongibson/32/267_2.png) [@carltongibson](https://forum.djangoproject.com/u/carltongibson)\
**Post date:** [February 24, 2023, 9:34am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/18 "2023-02-24T09:34:23Z")

</div>

OK, great. I shall take another look next week now. Good effort!

The **other** part of this project is bringing in the effort to modernise the request object. (We want to do both in the scope of the same release, 5.0) That **should** be a case of updating @adamchainz’ original PR. If you wanted to browse that and have a think about it, that would be worth your time.

Check-out the django-upgrade project. Creating _updaters_ to help people migrate their code would be a nice touch, and interesting (from the Python `ast` point of view).

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [February 24, 2023, 10:11am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/19 "2023-02-24T10:11:31Z")

</div>

Yes, will go through both the things next week. Thank you for these references!

---

<div class="post-metadata">

**Author:** ![udbhavsomani](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/udbhavsomani/32/12210_2.png) [@udbhavsomani](https://forum.djangoproject.com/u/udbhavsomani)\
**Post date:** [March 2, 2023, 8:25am UTC](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865/20 "2023-03-02T08:25:34Z")

</div>

Hi @carltongibson!  
I went through a bunch of threads (github + mailing list + tracker). I am in the process of drafting my GSoC proposal for the same.  
Just wanted your inputs for the same:

- Adding the `request.query_params` property and refining the request object further.
- Create a new parser class that will check for the `content-type` of the request object and try to return parsed python object(s).
  - This can be activated by adding it as a middleware? can be there by default as well.

- A fallback to the default parser if this middleware class does not return any compatible parser.
- Adding this migration to the django-upgrade project

These are the things I’m currently looking to get done during the project term. Please add on to anything that I might have missed.

[Next page](https://forum.djangoproject.com/t/discussion-on-configurable-content-type-parsing-project/18865.md?page=2)