Rename documentation files from `.txt` to `.rst`

Django Internals Documentation
1

13 years ago, Ticket #18448 proposed renaming the docs files from .txt to .rst, so editors and GitHub would automatically use reStructuredText syntax highlighting and other features. @jacobian closed it as wontfix it at the time.

In the last two weeks, @mlissner and @medmunds have both commented challenging the resolution. I agree with them, so it sounds like we have some impetus go ahead with the rename.

There would be some temporary disruption, such as for docs backports by the Fellows (@sarahboyce @nessita ). But I think it would be worth it in the end.

Any opinions?

4 Likes
2

I agree that this is causing ongoing pain. I am in favour of a one-time churn here to improve the experience long-term.

2 Likes
3

I agree on renaming it.

This was discussed in October last year in the discord with mixed opinions.
https://discord.com/channels/856567261900832808/859997770274045954/1298016328774582396

In November there was discussion of an action to detect documentation length issues.
https://discord.com/channels/856567261900832808/859997770274045954/1308931342867759206

For this last one, there’s a github action that I found, however it expects .rst files. I created an issue to support .txt files but never got a reply.

4

To share the points from @felixxm on the Discord, in case anyone doesn’t have access:

  • this needs investigation as to whether this would break doc translations
  • this will make checking git history more complicated as --follow works only for a single file

The git history impact is the main concern, unless there is a solution for this, I don’t think this can be progressed (or is worth progressing).

5

Thank you, Alex, for sharing the Discord link. I was just about to reference the recent discussion on this topic in Discord as you just did.

From my reading of the summary, there seems to be a strong sentiment that renaming the files could cause more problems than it would solve, particularly when it comes to Git history and Fellows/reviewers/triagers workflow. Given these potential complications, I’m inclined to think that changing the file extension may not be worth it.

One proposed middle ground was allowing a mixture of .txt and .rst files, but I’m strongly against that approach, as I feel it creates inconsistency that could complicate things more than it helps. In a way this approach would be the worst of the two worlds. Overall, I lean towards not progressing this change unless there’s a more substantial benefit to offset the costs.

Side note: I don't think Discord is that useful when referencing public conversations, this is a separated topic but because I have had troubles reading past Discord conversations, I'm extracting the relevant bits and adding them below for posterity. Expand!

Thibaud Colas — 10/21/24, 5:13 PM

Does anyone know why Django uses .txt for its reStructuredText docs? I find myself wondering if there’s a reason why. I could use more syntax highlighting

Carlton Gibson — 10/22/24, 5:53 AM

No idea the origin: Dawn of time. You should be able to set language-file associations in your editor (per project hopefully). I’m not against you suggesting changing it to .rst but I’m not opening the batting there either

Thibaud Colas — 10/22/24, 5:54 AM

I have, just can’t do that in GitHub, and I’m hesitant to suggest this workaround for new contributors I help with docs changes

Alex — 10/22/24, 6:17 AM

Changing it makes the future easier for everyone since we use the “standard” file extension, I think the question is if the possible merge conflicts for existing branches when the change happens are worth it.

Baptiste — 10/22/24, 6:32 AM

Wouldn’t git be able to handle it if it’s a straight up rename?

Thibaud Colas — 10/22/24, 6:36 AM

yes IIRC straight-up renames should be fine, just needs some finnesse if we had to do renames and then updating file references(?)

Mariusz Felisiak — 10/22/24, 6:38 AM

I wonder if this will not break doc translations

and it will make checking git history a bit more complicated

as --follow works only for a single file

I don’t think it’s worth doing

CodenameTim (he/him) — 10/22/24, 7:32 AM

git diff should handle renames, unless I’m misunderstanding your use case.

Mariusz Felisiak — 10/22/24, 7:45 AM

I’m taking about git log or git blame

It’s a every day job to dig into the source of changes in the Django repository

Marijke (Mah-Rye-Kuh) — 10/22/24, 7:48 AM

A file’s Git history doesn’t move with a rename. You would have to rewrite all the historical commits if you want to retain them

Mariusz Felisiak — 10/22/24, 7:49 AM

For example:

$ git log django/db/backends/base/__init__.py

commit 28308078f397d1de36fd0da417ac7da2544ba12d
Author: Tim Graham <timograham@gmail.com>
Date:   Mon Jan 12 15:20:40 2015 -0500

    Fixed #22603 -- Reorganized classes in django.db.backends.

Tim renamed this file so you have to dig deeper with --follow

git log --follow django/db/backends/base/__init__.py
commit 28308078f397d1de36fd0da417ac7da2544ba12d
Author: Tim Graham <timograham@gmail.com>
Date:   Mon Jan 12 15:20:40 2015 -0500

    Fixed #22603 -- Reorganized classes in django.db.backends.

commit ed114e15106192b22ebb78ef5bf5bce72b419d13
Author: Adrian Holovaty <adrian@holovaty.com>
Date:   Wed Jul 13 01:25:57 2005 +0000

    Imported Django from private SVN repository (created from r. 8825)
    
    
    git-svn-id: http://code.djangoproject.com/svn/django/trunk@3 bcc190cf-cafb-0310-a4f2-bffc1f526a3

Marijke (Mah-Rye-Kuh) — 10/22/24, 7:51 AM

Do the SVN logs still exist somewhere? The git-svn-id link gives me a 404 on Trac.

This is akin to programmer archeology.

Mariusz Felisiak — 10/22/24, 7:52 AM

renaming all doc files will force Fellows (and others like me) to do this every single time

Mariusz Felisiak — 10/22/24, 7:52 AM

As far as I’m aware you would have to have copy of the private SVN repository

ed114e15106192b22ebb78ef5bf5bce72b419d13 this is like a holy grail or big bang

the beginning of everything

Marijke (Mah-Rye-Kuh) — 10/22/24, 7:54 AM

These days, I am in the fortunate mental state that I often forget that SVN existed. I wouldn’t even know how it works, anymore

Marijke (Mah-Rye-Kuh) — 10/22/24, 7:54 AM

Time to slap a theme song on it

But in all seriousness, since I was looking at some code yesterday that predates the Git repo, I found it unfortunate that I couldn’t find the reasoning behind some of the lines in the code. That is under the assumption that the SVN commits would have a descriptive message, but the context usually helps.

(Not to derail Thibaud’s original question about why the RST files use the .txt extension and whether we still want that)

Ben — 10/22/24, 8:01 AM

You can centrally mark certain commits to be excluded from git blame I believe, if that helps

Mariusz Felisiak — 10/22/24, 8:03 AM

It won’t help

as it’s not about having less history

but about tracking to the source of specific changes

CodenameTim (he/him) — 10/22/24, 8:05 AM

Does it git log -- follow do what we’re missing? Git - git-log Documentation

Ah single file

Boo

Marijke (Mah-Rye-Kuh) — 10/22/24, 8:05 AM

You can even configure your Git to automatically follow blames, but it has some performance implications, and you don’t control how other people configure their system.

IMO, I would say “If it ain’t broken, don’t fix it” in this case.

CodenameTim (he/him) — 10/22/24, 8:06 AM

Configuration definitely feels like something django can control

CodenameTim (he/him) — 10/22/24, 8:07 AM

But the original case (rst files using txt file extensions confusing new contributors) is arguably broken.

Marijke (Mah-Rye-Kuh) — 10/22/24, 8:10 AM

That’s a minefield. You cannot control someone’s global .gitconfig (thank goodness), and arbitrarily messing with default behavior is confusing. It’s the same with things like .git-blame-ignore-revs ; I occasionally use it and put instructions in a README for people on activating it, but it’s all optional.

Marijke (Mah-Rye-Kuh) — 10/22/24, 8:11 AM

Fair. Someone is going to lose somewhere. But I think the impact of converting from .txt to .rst is much bigger than keeping it the way it is now. Of course, that doesn’t mean we shouldn’t consider it.

Thibaud Colas — 10/22/24, 8:14 AM

Going through the commit history, can confirm it’s been .txt since the dawn of times (here’s the first ever docs file from what I can see). So I’ll just assume there wasn’t a .rst back then and that’s about it as far as the decision to use .txt .

I’m not going to suggest we’d want to change this, just interested in the rationale when I explain this to others. It looks like this to configure VS Code to treat the files as reStructuredText if anyone needs it:

"files.associations": {
    "**/docs/**/*.txt": "restructuredtext"
  }

Marijke (Mah-Rye-Kuh) — 10/22/24, 8:20 AM

If anyone needs a reminder, I listed a variant of this for Django on my blog: Quick tip: Set folder-based file associations in Visual Studio Code / Marijke Luttekes

Thibaud Colas — 10/22/24, 8:23 AM

oh if anyone did want to suggest that .rst change, it does seem like all of the above is only pain related to existing files moving over. So new docs could be authored as .rst it seems? Sphinx has good support for mixed file extensions / formats.

nessita — 10/22/24, 3:05 PM

I’m torn and terrified by having some with one extension, some with other extension

MY EYES MY EYES

I personally think it’s a bad idea to have a mixture of file extensions (txt and rst), because in that scenario, you have to explain why txt and WHY a little bit of both. To me is the worst of the two worlds.

nessita — 10/22/24, 3:16 PM

I would be open to consider a moving away from txt files but in that case I would advocate to migrate to markdown

so we do a proper disruptive thingy worth messing up with the history (but please note I’m not saying we should)

Baptiste — 10/22/24, 4:09 PM

Maybe host the documentation on Github wiki (sorry, that was a bad joke)

Marijke (Mah-Rye-Kuh) — 10/22/24, 4:37 PM

Was it a joke?

nessita — 10/22/24, 4:37 PM

since we are talking docs, am I missing something or these definitions for “attributes” inside class definition should have their description indented one level in? https://github.com/django/django/blame/main/docs/ref/contrib/gis/gdal.txt#L842 (cc/ @Mariusz Felisiak maybe you know how’s the correct indentation for these?)

Marijke (Mah-Rye-Kuh) — 10/22/24, 4:38 PM

I foresee the migration of the docs to Markdown files as Natalia’s Magnus Opus as a Fellow

nessita — 10/22/24, 4:38 PM

Please note that I wasn’t necessarily proposing that migration!

I said that IF we will consider the file extension rename, THEN we should do the whole mic drop properly

Marijke (Mah-Rye-Kuh) — 10/22/24, 4:39 PM

Just teasing you Although I wouldn’t be against it; I don’t like RST haha.

But yeah, if you do something drastic, better do it well the first time.

Baptiste — 10/22/24, 4:42 PM

You mean “should” as in “otherwise sphinx doesn’t parse them right”, or as in “according to Django’s styleguide”?

nessita — 10/22/24, 4:48 PM

Neither, as in “should to be semantically correct”

Ülgen — 10/22/24, 9:33 PM

Wouldn’t be more interesting than trac - github issues migration

Carlton Gibson — 10/23/24, 6:00 AM

Wowser that last 60 messages or so is quite painful

2 Likes
6

My comment about this was four years ago, not two weeks, but I’ll stand by my point that syntax highlighting is useful.

Sounds like the question is:

  1. Syntax highlighting now and forevermore is a nice thing.

  2. But this will break git history (though this will get better over time as old changes become less relevant).

So, which is more important? To me, an occasional tweaker of documentation that rarely touches reStructured text, the syntax highlighting is much more important than the git history. I imagine when reviewing PRs that’s true too.

How often do folks dig into the git history of documentation files? Seems sort of niche to me, particularly for documentation files, but perhaps my experience is limited?

2 Likes
7

If it’s just about syntax highlighting on Github, can that be solved with a config change like the one you did here? Or am I missing something?

8

That was my thought too. (And what precipitated bringing this up again.) But GitHub has a preview rendering bug that makes .gitattributes linguist overrides unworkable for text markup types.

Also, it’s not just about syntax highlighting on GitHub (though that would be a useful step). It’s about editor support in general.

2 Likes
9

As a Django Fellow, I can say that day-to-day work relies heavily on git history, far more than on syntax highlighting. Speaking personally, I rarely use GitHub’s preview or syntax highlighting when reviewing. Instead, I always branch and work locally because a thorough, in-depth review requires more than what the GitHub UI provides. The first example that comes to mind is having to disable or change code to ensure the provided tests are correct.

It may not be immediately obvious, but in Django’s case, reviewing documentation often involves cross-checking against code, existing practices, docstrings, and previous documentation. When inconsistencies arise, we frequently rely on git history to determine whether a change was intentional or a mistake. This is a routine part of maintaining consistency and accuracy.

Unfortunately, this won’t improve over time, at least not within a reasonable timeframe. If you check the history of most documentation files (outside of the releases folder), you’ll find commit records spanning 15 years or more. Django values long-term stability, so both code and documentation are maintained with longevity in mind. We rarely remove or replace things outright, meaning history remains relevant for much longer than one might expect.

I’m not in favor of making an invasive change to workaround an bug in a platform that is younger :baby: than the framework we love. I know Mike said this is not the only reason, so more on that below :-).

Could you share more details on this? Are there specific editors that aren’t handling .txt as .rst well? It would be helpful to gather more concrete examples or data points to better understand the impact across different environments.

10

In PyCharm, you have to manually override the file type for each individual .txt file, every time you open it. Or change your global .txt type to reStructuredText, which gets confusing for plain text files in other projects. (JetBrains users can—and should!—vote for overriding file types per project and by directory pattern.)

GitHub’s quick editor doesn’t seem to have any way to control the file type, other than by file extension.

It looks like VS Code has a config option:

With all of these, you have to first know that the .txt files are actually reStructuredText. And that that’s a markup format with editor support you might want to enable. (And it’s different from Markdown.) It’s maybe not a huge barrier, but I’m guessing it’s one more stumbling block for new contributors.

11

(Having said all of that, I’m -1 on anything that makes the Django Fellows’ work more difficult.)

2 Likes
12

Enabling log.follow in your Git config makes this automatic:

$ git config --global log.follow true

I recommend setting this in my Git book. As far as I know, there aren’t any drawbacks to setting it globally. It’s just one of many options that Git didn’t enable by default due to backwards compatibility concerns.

By the way, GitHub’s commit history doesn’t follow renames but presents a button to link through the rename, for example:

image
image2466Ă—900 178 KB

So at least it’s just a click.

Marijke wrote in the Discord conversation:

You can even configure your Git to automatically follow blames,

git blame isn’t a concern either since it automatically follows file renames.

I think she may have been hinting at git blame -C, which makes it follow chunks of code moving between files.

Same for me in Zed, I had to set in my project settings:

{
  "file_types": {
    "reST": ["txt"]
  }
}

Right, that’s fair, but did you configure your editor to get reST highlighting or other features?

This is a good question. As far as I can see, the translation files do not specify the filename of their source document, e.g. this one. In fact, they’re merged across many documents already, like all of the ref docs have their translated strings in one giant ref.po file.

So, we might be good? Obviously needs testing.

I’m still in favour of the change, and I think most concerns can be mitigated. But whatever the consensus here, I’m more happy that we’ve discussed it properly! Thanks everyone for taking time here.

6 Likes
13

When I say “editor support” for reStructuredText, I’m talking about not just syntax coloring, but also things like:

  • autocompleting reST directives and Sphinx references
  • inline linting and error detection (e.g., PyCharm’s “inspections”)
  • a rendering preview panel
  • applying Python highlighting, linting, autocompletion, and formatting in embedded code blocks

One concrete example: Just this week, I committed a syntax error to a Django docs file—one not caught by a local docs build or Django CI. (Mismatched double backticks. I noticed it only because I happened to check the ReadTheDocs PR preview and saw something looked off.) If I had overridden PyCharm to use reST mode, the error would have stood out. But I tend not to bother for minor docs edits, because it’s just enough of a hassle that it breaks my flow.

Thank you for opening the discussion. It surfaced a bunch of concerns that weren’t captured in the ticket.

(Incidentally, I’ve reported the GitHub bug: Preview rendering gets confused by linguist-language override · community · Discussion #154413 · GitHub. But I’m not holding my breath for a fix.)

14

Yes, exactly. I’m using Esbonio, an LSP for reST. It provides a lot of those same features for any editor that supports LSPs.

1 Like
15

@adamchainz Good question. I’m old school and I don’t use any reST highlighting, so that’s probably why I don’t mind it. I treat docs like plain text files. The only thing I really need is the 79cols ruler :slightly_smiling_face:.