Trac migration informational DEP - Feedback required

[tom]

@emma

But would you still feel the same way about github, gitlab or another modern issue tracker?

I’m not 100% sure. There are a couple of differences I think:

First, I think it’s a bit much to expect people to keep up with an issue tracker. The idea of bringing something to the list / forum is that it will get more eyes on it because it’s a bit more of a place people browse, maybe because the volume is lower.

The other thing is that I think closed issues are a bit harder to find. So if you close the issue but allow conversation to continue to build consensus there, it’s a bit harder to find. On the other hand, an open issue is often an indicator that it should / could be worked on. You can probably get around this with good labelling though. Conversely, forum threads don’t really get closed.

[KenWhitesell]

@tom

Conversely, forum threads don’t really get closed.

To be quite pedantic here, they can be - and a few have, when they have drifted too far off-base or have become too contentious. (I’m only pointing this out because it is a tool in the moderator’s toolbox that can be used when needed.)

[tom]

@KenWhitesell

they can be - and a few have

Yep, I should have been more specific here. I only mean that issues are expected to be closed at some point, but forum posts are expected to not be closed (unless things go off the rails).

[doom]

@ryanhiebert

A better way might be to watch for stalled design decision needed tickets for a certain amount of time, and auto-close them after a certain amount of time with a message encouraging the next person to open it up again to the design decision needed state if they also restart the discussion on the forum.

Who is “people”? Because that implies anybody canreopen a closed ticket, but that is not true. Only maintainers can and from my experience in different repos, closed unsolved tickets get reopened extremely rarely. Instead, this causes multiple tickets to get opened because the existing one is already closed.

What is the harm to keep a ticket without design-decision open? Does age change that a design-decision is still needed?

Closed, in my understanding, means that the ticket does not need any further discussion, because it was solved, fixed or it was decided that no action is to be taken. Closing a ticket should convey that it does not require further conversion or action. Everything else should be left open, IMHO.

[doom]

@ryanhiebert

We can’t just automate a thread for every design decision needed ticket, because that’ll be far too many that nobody actually thinks are that important.

Quick note: GitHub let’s you move issues to discussions and vice versa, if that’s helpful here.

[tom]

@emma

More often than not, it’s the later, regardless of how deeply the organisation is attached to said process

One bit of friction I tend to get with those that are in favour of Trac is that it supports community triage and that we hold this as an important thing to safeguard.

So I went into a rabbit hole a bit.

I wanted to answer the question: How much community triage is there in reality, and is it really a good thing?

I got a CSV of the last 1K tickets from Trac. 1K turned out to be a great number because it’s just over a year’s worth of tickets, so it should serve as a good pulse check for where things are in the last year or so. I then ran some analysis on the tickets to figure out who triaged the ticket. I won’t go into details but I can put the code up somewhere if anyone is interested. I made a list of teams and put people into them. Anyone not in a team was put into the “Community” category. I then went through the tickets that were triaged by the community to see how well that went.

For team composition, some people are in multiple teams so I chose one in the following priority order:

• Fellows
• Former fellows
• T&R team
• Any other teams on the teams page
• Anything else (only the board ended up here)

Carlton and Mariusz are in the T&R team, but Mariusz was still a fellow for some of this period. So the T&R team should be a bit higher and former fellows a bit lower, but I didn’t want to put a ton of time into it. Another thing of note is that the SC is the new SC, not the one that existed for most of the period. But the only previous members of the SC to triage anything were also in other teams.

The results:

Team	Tickets triaged	Percentage
Total	1000	100
Fellows	479	47.9
Former fellows	268	26.8
Triage & Review	166	16.6
Community (no team)	39	3.9
Mergers	28	2.8
Untriaged	7	0.7
Accessibility	6	0.6
Ops	3	0.3
Security	2	0.2
Steering Council	1	0.1
Board	1	0.1

So I dug into the 39 from the community.

Fully 15 of these were people closing their own tickets after realising they got it wrong. This is great, but it isn’t community triage, and this can be done on other non-Trac systems.

4 of these were people incorrectly accepting their own ticket.

1 was accepted by the PR author after the ticket was made by a fellow. Fine, but not community triage either.

Another accepted their own ticket based on an accepted DEP, which is technically against the rules I suppose, but probably fine here, but still not community triage.

So 18 of the 39 don’t really count.

Of the 21 left that I would count:

5 were incorrectly triaged and needed someone else to step in and fix it.

3 hit an edge case with my code where the ticket was closed by someone from the T&R team, then incorrectly re-opened by someone else.

So out of the 1000 tickets, only 13 were correctly triaged by the community.

Of these, quite a handful were people accepting a ticket while assigning themselves. Which is probably fine, but I’m not sure it feels like what community triage is aiming for if you only triage things you want to also fix yourself.

Either way, we end up at a maximum of 1.3% well-handled community triage over the last year. I’m not sure that seems to be worth holding onto at all costs.

You might say that I am deflating the numbers here by including the T&R team since they were picked exactly they were doing this already. I’m not sure this is exactly the case though. I don’t think any of those people were chosen purely on their triage abilities. I think they were mostly picked for their review abilities. I say this because many of the T&R team don’t show up doing any triage at all, but every single one of them reviews PRs and I believe they’ve all made their own code contributions as well. So I’d be very happy to expend the team to more people, but I’m not sure we really need to allow everyone to triage.

[timgraham]

Thanks for the analysis, Tom. Correct me if I’m wrong, but it seems you only considered making a decision about an unreviewed ticket as triage. Triage is much more than that. As a new contributor years ago, I felt empowered that I could update a patch and remove flags like “patch needs improvement” or mark a ticket as “ready for checkin” after I gave it a thorough review.

Python’s documentation says, “When you have consistently shown the ability to properly help triage issues without guidance, you may request that you be given the “Triager” role on the issue tracker.” It’s not obvious to me how one can demonstrate the ability to triage issues when one cannot make such changes.

[tom]

@timgraham

Correct me if I’m wrong, but it seems you only considered making a decision about an unreviewed ticket as triage.

You’re absolutely right. I took a very conservative definition of triage: the thing you do first to sort the patient (ticket) into the right bucket. I didn’t consider much else to keep it simple.

@timgraham

Triage is much more than that. As a new contributor years ago, I felt empowered that I could update a patch and remove flags like “patch needs improvement” or mark a ticket as “ready for checkin” after I gave it a thorough review.

Yes, the way we currently do it, certainly. I think the specific example you give feels a bit toilsome to me. Really if you review something and request changes these types of things should ideally update automatically. As is, often people forget - especially after making requested changes - and their PR often languishes outside the review queue. Not very empowering to those that forget.

There are still a lot of places to feel empowered though. Leaving a review without setting flags manually should achieve something similar. And of course there is the empowerment of writing an accepted ticket, having a PR merged, etc.

Python’s documentation says, “When you have consistently shown the ability to properly help triage issues without guidance, you may request that you be given the “Triager” role on the issue tracker.” It’s not obvious to me how one can demonstrate the ability to triage issues when one cannot make such changes.

They are also using GitHub so we can probably learn a lot from how Python does things. There are a lot of interesting things to read here. One is that they say in order to effectively triage issues you need to have already worked on the codebase:

Do realize, though, that experience working on Python is needed in order to effectively help triage.

Another is that people with contributions can be asked to be added to the triage team, you can ask a core dev (not sure what that would be for us, possibly someone already in the T&R team) to join, and if accepted they will offer some mentorship. Perhaps this is all a bit heavy handed for Django’s needs, but it shows a clear path in.

Another thing I think is important: triage isn’t just setting flags on tickets. Some of the more time-consuming parts of triage is doing things like reproducing bugs, running git bisect to find where something regressed, building MREs, reviewing code (if you want to expand triage to that). These things are always open to anyone.

[Lily-Foote]

Thanks for doing this analysis Tom!

I don’t think any of those people were chosen purely on their triage abilities. I think they were mostly picked for their review abilities. I say this because many of the T&R team don’t show up doing any triage at all, but every single one of them reviews PRs and I believe they’ve all made their own code contributions as well.

I’m a member of the Triage and Review team and I’m pretty sure in my case that’s based on review rather than triage, for the most part. For those times I’ve been involved in triage it’s been as a navigator for Djangonaut space or as a interested reader of a forum post.

Either way, we end up at a maximum of 1.3% well-handled community triage over the last year. I’m not sure that seems to be worth holding onto at all costs.

I think this is right. I think people good at triage are likely to put the results of their triage work in a comment on the issue, even if they don’t have permissions to update flags or close the issue. If we move to another platform we can keep an eye out for people to trust with these permissions. Depending on the permissions, we might be able to be very open with giving them - someone assigning the wrong labels or closing an issue incorrectly is easily fixed.