Switch to Playwright tests for integration testing

Feature Request Link:
Use Playwright for integration testing · Issue #13 · django/new-features · GitHub

Project Description:

This project aims to modernize Django’s integration testing by introducing Playwright as an alternative to Selenium. The work includes integrating Playwright with Django’s test framework, migrating existing browser-based tests, and ensuring compatibility with Django’s CI infrastructure. The goal is to improve test reliability, performance, and developer experience for Django core contributors.

Attaching draft PR created in my fork: ​https://github.com/varunkasyap/django/pull/7/

Trac ticket: #37154 (Switch to Playwright tests for integration testing) – Django

My plan is to update Django’s browser testing setup by moving from
Selenium to Playwright

Why Playwright is Better
● Playwright doesn’t need external WebDrivers
● Playwright has built-in auto-waiting for actions.
● Playwright is much simpler to write and read, lowering the cognitive load for
contributors.
● We can delete custom helper functions that we needed for Selenium.

Project Size: As of today, there are 106 selenium test cases in main branch that need to be converted.

Project Goals:

  1. Modernize the testing infrastructure.
  2. Rewrite existing tests
  3. Boost CI execution performance.
  4. Enhance test reliability.
  5. Improve the developer experience with better tools

The project preserves Django’s testing architecture and only replaces the browser
automation layer (Selenium to Playwright).

10 Likes

Thanks for posting @varunkasyap. This is an exciting project :star_struck:

2 Likes

This would indeed be very beneficial. However, the devil is in the details and it might take a considerable amount of time to rewrite all the tests.

1 Like

Great idea! :ok_hand: Agree with Jacob about the effort, though.

1 Like

Summary of progress:

  • Created base classes: PlaywrightTestCase,
    AdminPlaywrightTestCase, and AdminWidgetPlaywrightTestCase.
  • Deleted one liner helpers that is not really needed in Playwright, example: select_option(), select_option(), is_disabled()
  • Added the --playwright flag in tests/runtests.py
  • Added Playwright to the test requirements and set up the CI workflows
  • Translated 55 tests out of roughly 106 :tada: half way
    Draft PR: https://github.com/varunkasyap/django/pull/7/

Design Discussion:

Even though the translated tests use self.assertEqual(), some waiting is still needed, such as wait_for_load_state() or wait_for(), because a few tests are flaky.

I am leaning towards using Playwright’s expect() since it provides built-in auto-waiting and retrying. Also we could use inbuilt helpers in expect(). However, I don’t think expect() is needed for every test case. For example, tests without page navigation or tests that only verify backend/database changes can continue using self.assertEqual().

My concern is that having both approaches in the same test file may not be contributor friendly. I am also slightly concerned about importing expect in every test case (from playwright.sync_api import expect).

Another option could be to import expect at the top level, (this could be improved in the future with Python 3.15’s lazy imports) but I’m not sure whether ImportError handling is necessary, considering Playwright is already listed in our requirements and expect is available in playwright.sync_api.

2 Likes

Nice progress! You’ve outlined the reasons to use or not use expect(), and I don’t expect that to be too confusing for contributors. About the imports, what about consolidating them in one place in an instance method on the base class? Then test methods would call self.expect().

1 Like

When translating admin_views.test_skip_link_to_content.SeleniumTests, I have realized Tab button works differently in WebKit.

Safari User Guide says: Generic Tab button do not highlight clickable items. Where as Option + Tab does.

Action Keyboard Shortcut
Highlight the next field or pop-up menu on a web page Tab
Highlight the next field, pop-up menu, or clickable item (such as a link) on a web page Option + Tab

Reference: Safari documentation

So, the test is failing in WebKit browser. May be we could either tweak the test or can skip test on WebKit.

1 Like

In Selenium, the UI elements are mostly located by class, id, CSS, or CSS selector. But Playwright suggests using built-in API locators instead.

Example:

self.page.locator('input[value="Save"]')

should be

self.page.get_by_role("button", name="Save")

Playwright locators documentation

Playwright locators look like:

Why use such web-first locators?

  • Resistance to future UI logic changes (less flaky)
  • It feels like accessibility verification as well.
  • Better test readability

But,

  • Needs rework on almost every test.
  • Language translation issues may arise.
1 Like

Test: admin_views.tests.SeleniumTests.test_child_popup_not_closed_when_parent_minimized

This Selenium test uses selenium.minimize_window() to minimize the browser window, whereas Playwright does not seem to have an API to minimize the browser.

We could use a CDP session for this, but that would be a Chromium-only solution, and we would need to skip this test on non-Chromium browsers.

Tests affected: 1

For the web first locators, I would suggest that this is out of scope for the migration
We could chose to rework tests that do not involve testing translations, however tests decorated with @screenshot_cases(["rtl"]) are testing a translated page and we add this decorator to existing tests fairly frequently. Therefore, it is nice if the tests remain translation friendly

I agree web first locators test accessibility better but we can do any updates here aligned with accessibility testing tickets

3 Likes

Created draft PR: Fixed #37154 -- Switched from Selenium to Playwright for integration testing. by varunkasyap · Pull Request #21596 · django/django · GitHub

Feedbacks are really appreciated.

1 Like

The current Selenium setup uses headed mode by default and --headless option for headless mode.

I’d like to propose making headless the default and introducing --headed for headed mode. I believe Playwright uses headless mode by default. We could also add a --slowmo option for slow-motion execution and --debug for Playwright Inspector. else we could document about PWDEBUG=1 environment variable to launch tests in debug mode.

Any thoughts?

2 Likes
  1. “Making headless the default and introducing --headed for headed mode” - +1, I think most people want to run the tests in headless
  2. “Add a --slowmo option for slow-motion execution” - +1 (I have found this useful in the past using playwright tests, however this is a “nice to have” that shouldn’t block any existing work)
  3. “Add --debug for Playwright Inspector” - 1, partly because I don’t have experience of the value and I feel that this option would be confusing for the usual tests
  4. “Document about PWDEBUG=1 environment variable to launch tests in debug mode” - +0, if this is a useful option it is worth documenting but I do not have experience using myself
2 Likes

I see self.assertGreater() equivalent is not available in playwright expect helpers. something like this helps:
Instead of “count > 1”, assert the 2nd item exists. For example:

self.expect(pages.nth(1)).to_be_attached()   # at least 2

instead of

self.assertGreater(pages.count(), 1)

What do you think, this is a good idea?

also, no helper in expect to do self.assertLess() work.

I’ve been using Playwright for a while now, and it’s been much less painful than Selenium for browser tests. The auto-waiting alone removes a lot of the boilerplate. Curious to see how the migration goes and whether there are any tricky cases in Django’s test suite.