testing with self.assertContains(..., html=True) is messy

I am developing in Django for many years and for some time now I feel that using self.assertContains(response, ..., html=True) is difficult to work with.
The main reason I see is that it is very difficult to find what happened when the assertion fails.
The typical output looks like this:

======================================================================
FAIL: test_change_get (blenderhub.apps.evaluation.tests.test_admin.RatingLastChangeReportAdminTestCase.test_change_get)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/project/blenderhub/apps/evaluation/tests/test_admin.py", line 211, in test_change_get
    self.assertContains(
AssertionError: False is not true : Couldn't find '<div class="form-row field-rating">    <div>        <div class="flex-container">            <label class="required" for="id_rating">Rating:</label>            <input type="text" name="rating" value="4" required id="id_rating" class="vForeignKeyRawIdAdminField">            <a href="/bksecretadmin/evaluation/rating/?_to_field=id" class="related-lookup" id="lookup_id_rating" title="Lookup"></a>            <strong>                <a href="/bksecretadmin/evaluation/rating/4/change/">                    Rating object (4)                </a>            </strong>        </div>    </div></div>' in the following response
b'<!DOCTYPE html>\n\n<html lang="en-us" dir="ltr">\n<head> .... </body>\n</html>\n'

Where the ... represents whole page of HTML, extremely long and unclear. Very difficult to find where the asserted HTML element should have been and what changed.

Often the asserted HTML changes after some update with only minor (e.g. class) change. I would like to see what changed on the first glance and just change the assertion code in such case, but with assertContains this could result in difficult search.

For this reason I developed django-assert-element application: assert-element · PyPI

With this code:

from assert_element import AssertElementMixin

class MyTestCase(AssertElementMixin, TestCase):
    def test_something(self):
        response = self.client.get(address)
        self.assertElementContains(
            '<html><div id="my-div">Myy div</div></html>',
            'div[id="my-div"]',
            '<div id="my-div">My div</div>',
        )

It works like this:

  1. It extracts the element from response by xpath (div with id = my-div)
  2. If the element is not found or it is found multiple times, it fails with error
  3. It normalizes (heuristically) the HTML of both extracted and asserted HTML pieces. It purposefully adds a lot of newlines into the HTML code to make the comparison easier. It also normalizes the whitespaces.
  4. It compares the two with assertEqual which in case of failure outputs nice diff.

The output in the case of the above example looks like this:

======================================================================
FAIL: test_element_differs (tests.test_models.MyTestCase.test_element_differs)
Element not found raises Exception
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/petr/soubory/programovani/blenderkit/django-assert-element/assert_element/tests/test_models.py", line 53, in test_element_differs
    self.assertElementContains(
  File "/home/petr/soubory/programovani/blenderkit/django-assert-element/assert_element/assert_element/assert_element.py", line 58, in assertElementContains
    self.assertEqual(element_txt, soup_1_txt)
AssertionError: '<div\n id="my-div"\n>\n Myy div \n</div>' != '<div\n id="my-div"\n>\n My div \n</div>'
  <div
   id="my-div"
  >
-  Myy div 
?    -
+  My div 
  </div>


Which is much cleaner.

Although I am quite happy with django-assert-element, I feel that all Django users should benefit from testing function similar to this.
What do you think? Shouldn’t Django contain some tools for more advanced HTML testing?

2 Likes

Hey @PetrDlouhy thanks for sharing.

I think there’s plenty of room for improvements to the error messages from the assertion helpers!

In your example here something more intelligent by default and then the verbose options for the full output would make sense. (Devil is in the details of course but generally +1)

1 Like

Yeah, this kind of improvement would be very nice. Generally +1 from me too!

I presume this would affect assertInHTML too.

Very nice I like it +1 :heart:

I always appreciate anything that eases the burden of deciphering failures.

We do something similar in iommi with verify_html: iommi/tests/helpers.py at 0e9e80b63b587d27d73a51b7e46e611bfeeeab7c · iommirocks/iommi · GitHub

One issue I see with your package is that it looks like it’d be awkward to use if you’re using pytest.