Proposal: Clarify ticket statuses and queues

[ontowhee]

Hello! I’ve been digging into Django’s Triage Workflow and Trac a lot lately, and I would like to gather your thoughts on a proposal to reduce the confusion with ticket statuses and ticket queues.

Proposal

Would it help to have a separate field to provide clarity for the “big gray area” of the Accepted stage? This field may be called “Solution Status” or something similar, and it can consist of 3 values:

• Needs Patch
• Needs Review
• Needs Improvement

The logic for the Solution Status would follow the definitions outlined in the Triaging Tickets documentation:

• Needs Patch = Accepted
• Needs Review = Accepted + Has Patch
• Needs Improvement = Accepted + Has Patch + {Needs Docs | Needs Tests | Patch Needs Improvement}.

It could look something like this diagram, where the Triage Stages remain the same, and now the Accepted stage is accompanied by the Solution Status.

Changes to ticket description

There are a lot of fields in the ticket description. It takes some thought to figure out the combination of Triage Stage, Has Patch, Patch Needs Improvement, etc. to understand where the ticket stands.

The idea with the Solution Status is to abstract away the Has Patch and Patch Needs Improvement flags. Reducing the number of fields that determine the ticket status can improve the contributor experience.

Changes to ticket form

Currently, many contributors can be confused on how to modify the ticket status. They may not understand the meaning of “Has Patch” or the other boolean fields. It is also not clear what happens when those fields are modified and which queue the ticket ends up in.

In some cases, contributors will set the wrong values, in other cases they modify the wrong fields. This creates extra work for the Triage and Review Team members to manually revert the fields.

Trac has a plugin called DynamicFieldsPlugin. We can leverage it to only show the fields that are relevant for a given Triage Stage or Solution Status. For example:

• When Triage Stage != Accepted, hide Solution Status, Needs Docs, Needs Tests.

• When Triage Stage = Accepted, show Solution Status defaulted to “(unset)”. The form will raise an error if the Solution Status is submitted with “(unset)”.

• When Triage Stage = Accepted and Solution Status != Needs Improvement, hide Needs Docs and Needs Tests

• When Triage stage = Accepted and Solution Status = Needs Improvement, show Needs Docs and Needs Tests.

The dynamic fields can help contributors modify the correct fields.

Changes to query page

While experienced contributors have techniques for navigating the ticket queues (bookmarks, reports tab, dashboard, etc), new contributors or occasional contributors can feel lost.

The query page can be enhanced by including quick links for the Triage Stages and Solution Statuses. These quick links can be borrowed from the preset filters outlined in the Reports tab, but having them available on the query page can be very convenient. Contributors can get a good sense of Django’s Workflow, and they can also easily access the queues.

Code Changes & Data Migration

Adding the Solution Status field and applying dynamic form fields can be accomplished by configuring the trac.ini file. A proof of concept can be found in this PR: PoC: Add Solution Status Field by ontowhee · Pull Request #267 · django/code.djangoproject.com · GitHub

There will be a data migration on tracdb’s ticket_custom and ticket_change tables to calculate and store the value for the Solution Status field and to remove the Has Patch and Patch Needs Improvement fields.

Adding the quick links for the queues on the query page can be accomplished by creating a new CustomSubNavigationBar class that inherits from the Trac “Component”. Proof of concept in this PR: Added tabs for queues. by ontowhee · Pull Request #266 · django/code.djangoproject.com · GitHub

Thoughts?

Would changes similar to these be helpful to your workflow and your interactions with other contributors? I would love to hear your suggestions!

[jacobtylerwalls]

Thanks for including clear demos of these ideas. I like the quick links.

Solution status seems to duplicate the ticket-next-steps box underneath the description. That box is below the fold and easy to miss. One idea would be to move that box above or near the top of the ticket, just show a phrase for the solution status, and cabin the full description to an info tooltip.

[sarahboyce]

I think the quick links are an easy win which could be achieved without adding a new column (+1)
Note that I have heard a few people suggest this before and I do think this makes “going to the review queue” achievable with less clicks

In case anyone is not sure what the status box is that Jacob is referring to, I have pointed this out in the screenshot below

That we can tweak the existing helper to make this a bit more prominent is also a nice idea and tests out the logic of the “Solution status” without a change to the form or a data migration.

It is really hard to tell how common it is that someone has misunderstood the ticket columns and ticket status. If we have a way to gage that then we might know if just doing the above is enough or we should tweak the form

[CodenameTim]

@sarahboyce

It is really hard to tell how common it is that someone has misunderstood the ticket columns and ticket status. If we have a way to gage that then we might know if just doing the above is enough or we should tweak the form

Looking through the Django Discord, there have been six instances over the past 2 years where a person raised a concern that they were told about the needs improvement flag.

It looks like you also raised a concern a few years ago about making the review queue more obvious. I’d imagine having a status field that summarizes the state more intuitively in addition to making the review queue accessible would help.

[jacobtylerwalls]

@CodenameTim

I’d imagine having a status field that summarizes the state more intuitively in addition to making the review queue accessible would help.

Agree with the need for more & better hints, but in case not evident, the reason I mentioned moving the next steps box was that in my mind it’s already driven by a solution status state. If we restyle the box, hoist it above the fold, and reduce the text to just “Waiting for author” in bold, then authors will know we’re waiting for them. (If not, then we’re dealing with authors that only want to interact on GitHub, in which case to fix that we could tinker with the boilerplate in the PR template further rather than anything Trac-side, although at a certain point we need to ask potential contributors to make a small investigation into the norms of the project).

[CodenameTim]

I do like that idea of distilling and moving the next steps box.

What are your thoughts on improving the scenario of a new contributor trying to get their ticket into the review queue / back into the review queue? The status field would make it a single field they’d need to change which seems like it would help.

[jacobtylerwalls]

The status field would make it a single field they’d need to change which seems like it would help.

In most cases only one field needs to be set anyway (the original proposal here mainly discusses Patch Needs Improvement and Has Patch, which aren’t set together). The other flags: we’re only talking about potentially hiding them at times – which incidentally I do worry would only increase confusion later.

I don’t have data to back this up, but I have memories of contributors simply not knowing any of the following existed:

• a review queue
• ticket checkboxes
• responsibility to set ticket checkboxes themselves

I was suggesting face-lifting the next steps box could solve all of that in a much smaller-scoped set of changes, but I’m not the target audience for this, so happy to hear any other ideas!

[nanorepublica]

In terms of the target audience, next year’s GSoC hopeful applicants are already starting to pop up, who are a good sample of that target audience. We might be able to get some good qualitative data from them about this problem.

[ontowhee]

Thank you for sharing your perspectives and insights everyone!

I’ll summarize the suggestions and concerns raised so far regarding Solution Status field:

1. Concern about data migration
2. Concern about confusion if hiding/displaying fields dynamically
3. Suggestion to update ticket-next-steps box (reposition, restyle, rephrase)
4. Suggestion to gauge confusion with ticket statuses and ticket queues, including the suggestion to collect data from prospective GSoC applicants.

I’ll try to address the concerns and suggestions. Keep in mind as you read through these, that the real questions we’re trying to answer at this stage are:

1. Is there a real problem?
2. Is the problem worth addressing?
3. Are the suggested approaches heading in the right direction?

(1) Concern about data migration

Thanks Sarah for bringing up the concerns about the data migration!

There may be a possibility to test this out the data migration on a non-production site. Or if that is not possible, would it be acceptable to test it out on a local machine? My thought is to run the sql statement to show that the data is converted properly.

Here is a concrete example for the SQL statement that will allow us to calculate the Solution Status values for the existing tickets and insert a new entry to the ticket_custom table for the Solution Status.

insert into ticket_custom (ticket, name, value)
select
	t1.*,
	'solution_status',
	case
		when t1.stage='Accepted' and not t1.has_patch and not t1.needs_better_patch
		then 'Needs Patch'
		when t1.stage='Accepted' and t1.has_patch and not t1.needs_better_patch
		then 'Needs Review'
		when t1.stage='Accepted' and t1.has_patch and t1.needs_better_patch
		then 'Needs Improvement'
		else ''
	end as solution_status
from
(
	select
		t.ticket,
		t.custom_data->>'stage' as stage,
		(t.custom_data->>'has_patch')::bool as has_patch,
		(t.custom_data->>'needs_better_patch')::bool as needs_better_patch,
	from
	(
		select ticket, jsonb_object_agg(name, value) as custom_data from ticket_custom group by ticket
	) t
) t1
;

Notice how the needs_docs and needs_tests fields will not factor into the Solution Status value. This is assuming that the existing ticket fields are properly assigned. We can further improve this query to exclude the entries with empty strings from being stored in the ticket_custom table. There will be additional queries to clean up the boolean fields.

The ticket_change table is a bit more involved, and I will have to work out the details. It involves reconstructing the snapshots of ticket state for each time the ticket state was modified.

One of the motivations behind the original proposal, besides making it more intuitive on the front-facing side of Trac, is that there is another project that could build on top of the changes from this proposal. However, now that I think about it, it may make more sense to start with that other project. I was hoping it would be easier to start with the changes in this proposal, but it might be easier the other way around. Let me think this through.

If you have any other suggestions on how to verify the data migration in general, please do share them. Running manual inspection on both a non-production site and on a local machine is typically the approach I’ve used to verify data migrations.

(2) Concern about confusion if hiding/displaying fields dynamically

Let’s address this after we decide whether there is anything worthwhile to change, and how to approach that change. For now, let’s assume the form fields are static.

(3) Suggestion to update ticket-next-steps box

@jacobtylerwalls

Agree with the need for more & better hints, but in case not evident, the reason I mentioned moving the next steps box was that in my mind it’s already driven by a solution status state.

You are right, Jacob, the solution status in next steps box is handled in tickethacks.js.

@sarahboyce

That we can tweak the existing helper to make this a bit more prominent is also a nice idea and tests out the logic of the “Solution status” without a change to the form or a data migration.

How will this test out the logic of the “solution status”? From my understanding, the logic in tickethacks.js will mostly remain the same. We could have a new variable to hold the text for the solution status, and then render this text in the next steps box on lines 281-291. If the logic remains the same, what is being tested here?

In the original proposal, the value of the Solution Status field is independent from any other fields. The only logic involved would be in the data migration which initializes the Solution Status values from Has Patch and Patch Needs Improvement. The Has Patch and Patch Needs Improvement fields will then be deleted from the database.

Please let me know if something isn’t clear or if you don’t agree with this approach.

(4) Suggestion to collect data

@jacobtylerwalls

I don’t have data to back this up, but I have memories of contributors simply not knowing any of the following existed:

• a review queue
• ticket checkboxes
• responsibility to set ticket checkboxes themselves

Thank you for sharing these points, Jacob! Thank you Tim for searching discord to gauge the number of complaints. I think this information does point to a gap in the contributors’ understanding of the development process, even if it’s still not clear how to best address the problem.

@jacobtylerwalls

I was suggesting face-lifting the next steps box could solve all of that in a much smaller-scoped set of changes, but I’m not the target audience for this, so happy to hear any other ideas!

I think if you use Trac, the proposed changes will impact your workflow and your collaboration with contributors. Your concerns about it being more confusing are very valid, because it will impact how you communicate with others!

I agree that reducing the text to show the solution status is an improvement, and I am aligned with making this change. However, just as you may be skeptical of the impact of a new field, I am skeptical of the impact of face-lifting the next steps box. Again, neither are not based on any data, but let’s take a look at a side-by-side comparison below.

Would it help to take a step back from Trac for a moment, and imagine similar UI on a different application?

For example, let’s say there is a brand new To Do app. There are 3 lists that to-do items can belong to:

• To Do
• In Progress
• Done

You are tasked with moving an item from In Progress to Done. Unfortunately, there is no drag-and-drop feature.

One potential interface looks like this:

Another potential interface looks like this:

Does stepping away from Trac help put things into perspective a bit? Absolutely fine if it doesn’t. (If we go back to the context of Trac, to keep things simple, assume there is no problem with data migration, and assume the form fields are not dynamic.)

My personal opinion is, even if the UI/UX changes seem subtle, even if these changes do not address the gaps in the contributor’s awareness about ticket checkboxes or the responsibility to set ticket checkboxes themselves, or the need for a reviewer to explain how the ticket fields should be modified, I think we can try it out for a bit and see how it feels to use it, perhaps on a non-production site, if possible.

Survey

If it is necessary to run a survey, here is one idea. A google form survey with links to two videos:

• A) One video will use Trac with the next-steps box repositioned, restyled, and rephrased.
• B) The other video will use Trac with the Status Field and dynamic form fields. See video from PR #267. Or we can redo this video and have static form fields, so there are fewer variables that we are testing. Or maybe we want to test 3 videos? I’m brainstorming…

Additional survey questions could include:

• Which UI do you prefer? A or B?
• How frequently did you use Trac in the past year?
  • Never
  • Once a week
  • Once a month
  • Once a quarter
  • Once a year
  • *Note: Your answer to this question does not disqualify the validity of your feedback. We’re looking for trends among different groups of people.

For the “target audience” for this survey, I am of the opinion that anyone who uses Trac is in the target audience.

Please share your thoughts. I’m interested to know what level of effort is required to move things forward and improve Trac.

• From the To Do app mock-up, did you lean more towards having the boolean fields or the status field?
• Would it help to see video walkthroughs of the next-steps box face-lift vs. Solution Status field?
• Do we need to run a survey to A/B test video walkthroughs?
• Would it help to put PR #267 or the next-steps face-lift on a non-production site?
• How can we best verify the data migration?

[jacobtylerwalls]

Two answers quickly.

@ontowhee

Would it help to take a step back from Trac for a moment, and imagine similar UI on a different application?

If we were designing a tracker from scratch, I think having as few knobs as possible would be best. But we aren’t–I think we should articulate what we want and favor lighter changes to get there.

Do you have a plan for handling query strings containing these retired params? (Cool URIs don’t change.)

How will this test out the logic of the “solution status”?

I didn’t interpret this to mean it would be testing any code, but rather that this would test whether we could get the value proposition of “contributors become aware we are waiting for them” with less effort on our part.

Thanks for the follow-up here

[ontowhee]

Thank you for your input! I hadn’t considered maintaining the existing uri parameters at all.

@jacobtylerwalls

I didn’t interpret this to mean it would be testing any code, but rather that this would test whether we could get the value proposition of “contributors become aware we are waiting for them” with less effort on our part.

I misinterpreted this statement, thanks for clarifying! It makes sense now.

@jacobtylerwalls

I think we should articulate what we want and favor lighter changes to get there.

I was hoping to start small with this proposal. Let me put it aside for now and focus on the bigger picture.

I appreciate your feedback!