Django benchmarking project

Django Internals Mentorship
21

Hi @deepakdinesh1123 – great progress so far!

I’m really looking forward to seeing the the test harness work using Locust, that sounds most exciting.

I have a couple of small questions.

  • Timeline between your 30 July and 25 August targets is large. Maybe there’s an interim target too, maybe not?

  • If there is time it would be good to consider if we have benchmarks in the right area. There seems to be alot on queries (which is great), but do we have enough coverage on the core request/response cycle as it’s this area where even small changes will have a large impact. Like I say, this is only if there is time.

22

Since the target was to create a Django project and load test it using different servers, I had mentioned a large period in my timeline. If I finish it before July 30, I will immediately start working on the next target and then write the benchmarks for different areas of Django

23

Hi @deepakdinesh1123 (@smithdc1) :wave:

Sorry for the slow follow-up here — I was taken-up with other issues.

First off, really great progress so far!

Then, a couple of thoughts for the rest of the time.

Wrapping up ASV work

Let’s make sure this is in a really good state.

Ideally we want folks to be able to pick it up and keep working on it, and we want it running regularly, and so on.

It’s on my list for this week/next week to have another sit down with it, and so I’ll comment more concretely then, but I’d like us to make sure we keep enough focus on this part of the project to get it to a good state to hand-off at the end.

The Locust project

So let me show you some tickets:

There are more, but let’s start here. (Have a read, see what you think, let me know if/when you have questions.)

The task here is to get some idea of the performance impact of adding async signals, or allowing streaming responses under ASGI (with both the existing response classes, and new async specific ones).

So the task if to create a test project, with example views — for, say, a streaming response (#32798 there) — and run it under WSGI (gunicorn with 4 pre-forked workers, say) and then compare that to running it under ASGI with the proposed changes. For that ticket we can then make a decent judgement as to whether it’s too slow to recommend, or not.

For the signals ticket, probably just a list and detail view, and a few fixtures, and then hitting that with a decent amount of traffic to see the overhead.

Make sense?

Up and running with locust isn’t too hard, but again we’d want to add some breadcrumbs (as How-To docs, in a README is fine) for others to follow.

It should be:

  • Quite good fun.
  • Not too hard… — got to write the views and so on, but it’s basic Django in the main. (:grimacing:)
  • And a great resource for seeing how different options play out.

This second part is under-specified, so I’d focus on whether we can get concrete answers to those tickets there. Again, much better something small and complete than too broad but not finished.

I’d suggest to start, create a new repo, see how far you get quickly, and then reach out for input. I have thought some about this, but haven’t had the time to push it through, which is what’s great about GSoC, in that hopefully you do! :smile:

Given those thoughts, I don’t really mind where you want to focus. Again I think you’re doing super, and it’s a question of how best we can support you in the remaining time.

Thoughts?

Kind Regards,

Carlton

24

Hey @deepakdinesh1123 — One follow-up here…

See Update the load test · Issue #1765 · django/channels · GitHub — this is about the Channels project, but pulling that into scope for our load-testing harness would be worthwhile. It’s not been touched in a while, but you might want to reimplement with the latest versions, and then run locust against it. (This enables us to look at Channels but also compare bare(ish) ASGI to the overhead of Django’s ASGIHandler.)

Let me know if you’re going to, and I can make sure to offer assistance if you need it.

25

Sorry for the late response

I created a Django project and ran it with gunicorn and ASGI, I also created the respective dockerfiles.

I had a few questions

  1. Should I add them to the django-asv repo itself or add them to a new repo meant for load testing?
  2. Should I use different ASGI and WSGI servers such as daphne,uvicorn, uWSGI, mod_wsgi to loadtest every project?
  3. Should I add a workflow that can be triggered to perform load testing or add instructions on how to perform load testing locally?
26

Hey @deepakdinesh1123 — I’d start with a separate repo to begin. We can merge into a subfolder (say) later, but we need to pin down the final layout first. (We can do that this next month.)

So a project gives you both a wsgi.py and an asgi.py, so running against both seems reasonable. I’d stick to gunicorn to begin for WSGI (I don’t think there’s so much development in that space right now, so it’s not so interesting to compare servers), but Daphne, plus unicorn, (plus…) is certainly interesting in the ASGI space.

Begin by adding instructions to run it manually. We can add a workflow later. First goal is being able to add an endpoint, or a new settings file (to test different middleware say) and then run it against locust to see how it goes.

Good stuff. :+1:

27

Should I update the testproject in older versions on channels to the latest version and load test them using locust or should I create a new project?

28

Happy if you want to include it in the new test project. (We can then resolve that issue with “Moved load-testing to …”)

29

Hey @carltongibson - I have created a repository with a sample django project and I have also added the dockerfiles to run them with different servers. Please check it out and let me know the changes that I have to make to the directory structure or project or locustfile.

I have currently added only some basic parameters for load testing and will be modifying them later I will also look into storing the results of load testing.

1 Like
30

Hey @deepakdinesh1123 — nice start there! :+1:

Give me a few days to experiment with it.

One thing we could do is add a number of model instance (User and/or Group might do) and hit the /admin/ endpoints at various scales, with different numbers of gunicorn workers to see how adjusting that affects throughput. (≈Expected is that throughput increases until you reach no of cores/%-time-spent-blocking , but seeing that in graph form would make a cool demonstration.)

From there, rebasing the async signals branch, and doing the same to see the overhead would be the step towards resolving #32172 (Adapt signals to allow async handlers) – Django.

(Similar for other tickets)

Bar the 4.1, 4.0.7, and 3.2.15 releases next week, this is top of my list now, so I hope to give you the missing guidance on some of the loose ends next week too. :woman_juggling:

I submitted you midway review. A+ Great effort so far!

Kind Regards,

Carlton

31

While implementing this, I wrote a method in locust to log in a superuser using a POST request but am getting a csrf token missing error. I tried to mitigate this by using a custom login template and adding the csrf token tag there but I am still getting the same error.

Thank you so much.

32

Did you see this SO post? Django Load testing using locust giving Csrf token error - Stack Overflow — The suggestions there seem on the right line. Otherwise push it to a branch on GitHub and @carltongibson me there, I’ll take a look.

33

@carltongibson Thank you so much, I was able to add new models and hit the admin endpoints, I have also added the results.

1 Like
34

Progress Update

Currently Working on:

  • Updating Load test in django/channels

    • I bumped up the versions of the libraries in the test project and changed some code that had been deprecated
    • Fixed the docker and docker-compose files
    • Currently working on creating a test harness using locust and load testing

  • Adding Request-Response benchmarks

    • I have added the following request-response benchmarks
      • default middleware
      • HTTP methods
    • I will add more benchmarks in the upcoming weeks

Work in Upcoming weeks:

35

@carltongibson I have added the load testing for the ticket #33735 (Add asynchronous responses for use with an ASGI server) – Django in the branch here please take a look at it and tell me the changes I need to make.

I have not been able to work much this week as I had my semester exams which are ending this Saturday, I will make sure to pick up my pace from Sunday and finish all the things.

36

Super, thanks @deepakdinesh1123 — I’ll take a look this afternoon.

I have not been able to work much this week…

No stress. You’re doing great, and school is top-priority.
I’ll try and draft up a few points that I think should be top of the list — likely making sure docs/READMEs are good enough so people can follow along themselves if they want to.

:+1:

37

Hey @deepakdinesh1123 —

I’ve opened an issue to discuss:

Please take a look.

In the time we’ve got left I think it would be awesome to turn what you’ve done into a template that can be reused to create test projects. (It can’t just be one big project because of conflicting versions etc.)

Over the next few weeks I will play adding/creating a few examples.

At the same time we’ll get the ASV stuff merged/polished a bit. (I know you’re waiting on me there)
If you have spare cycles, porting a few more benchmarks is a good filler activity I guess.

Then I think you’ve got two really solid pieces of work you can point to at the end of the project, and a couple of blog posts you can write, and things you can point to on the portfolio. A nicely rounded GSoC I’d say :slightly_smiling_face:

38

@carltongibson I have added the channels-loadtesting repository here, Initially, I tried to migrate the existing project but ran into issues when migrating from channels v1.x so I created a new project.

Should I also add the benchmark.py and fabfile.py or should I just use Locust?

Currently, the Redis channel_layer and the in_memory channel layer are working correctly but I am having trouble with Postgres and Rabbitmq and am working on it. Please check out the repository and tell me the changes I need to make.

1 Like
39

Thanks @deepakdinesh1123 — Sorry for the slow reply, I’ve been ill this week.

I’ll take a look.

40

@carltongibson Hope you are doing well now

I have created my final work submission report using GitHub gist please tell me any changes that I need to make.

I will continue working on the template repository and channels load-testing project and I will also add some new request response benchmarks.

1 Like