Trust in Django & LLM generated code in core

Folks are increasingly using LLMs to generate PRs for Django.

We’re asking folks to declare that usage, and often they do, but I still see a lot of commits (not necessarily just in Django) that are clearly (at least largely) LLM generated but not so attributed, via e.g. co-authored-by lines in the commit message or any other mechanism.

I think we’ve just drifted into accepting such contributions without having had a clear community discussion about that, and whether, and to what extent, Django wants to accept LLM contributions, in core, and elsewhere.

@CodenameTim posted recently on the topic of Why use open source at all?:

I’ve also been rereading the recently updated 6th Edition of @wsvincent’s Django for Beginners.

As such, and given the developments in the last couple of years, I’ve been thinking about what Django provides for me, and why skill and expertise still matter, even with the new tools that are available.

Yesterday, we were recording Django Chat with @hakib. The topic of Django’s API story came up. What about if we just got Opus 5.5 to write one for us? was the question at one point.

Now @FarhanAliRaza has already done this: Django-Bolt — “Faster that FastAPI. Still Django” :slightly_smiling_face: — is a phenomenal project: a ridiculously fast Rust based Django API framework, written with significant LLM assistance, right Farhan? (I think I recall other projects in this space, but they’re not coming to mind immediately.)

Now, Django-Bolt is a great project, really exciting, but, I ask myself, how would I feel if Django just incorporated it, and declared that that was the long-awaited API story?

Or, to take another example, if we vibe coded a replacement for the ORM, how would I feel about that?

And I think the answer is that it would fundamentally undermine my faith in Django. It would cease to have a point, to adopt the dual of Tim’s argument.

I continue to use Django because it’s an absolutely solid and trusted foundation upon which I can do my work. The web problem is solved — we can add support for the new QUERY verb, and so on, but it’s done.

The interesting bit is what I do on top of Django. If I want to use LLMs there I can, but Django is the substrate: whatever mess the models come up with, Django is the foundation that lets me make sense of it, and stay in control. In this age of cheap code generation, it’s the known quality of the solid core that means something, not feature X, or Y, or Z, which I can have in moments just by asking for it.

I think there is a very real danger that we are drifting into a position where we incrementally undermine the quality and stability that constitutes Django’s value as the framework of choice, without having had the proper discussion about how to maintain that quality, and what limits we might draw on machine generated code where that might undermine it.

To wit, I think it’s long past when we should have had a bigger discussion about what our AI Policy should be. (I think we’ve somewhat been avoiding it, but I don’t see that as sustainable.)

It seems there are two schools: something like Twisted or Flask, which steer away, and then others, which seem to be accepting changes at ever faster rates.

My own view is something like this:

Whilst impressive, the output of even the latest models is deeply flawed, in ways that is often hard to specify. We all know it from LLM generated writing: well, so for code. It’s an empirical conjecture but, the kinds of issues I still see are largely unchanged over the last few years — they are better now, yes, but I’m not convinced the issues are going to disappear: these things generate next token, even in loops, and the lack of perspective seems fatal.

I see problems accumulate as LLM usage in a project increases. The ability to comprehend machine generated code diminishes, and the maintainability of that code diminishes with it.

In a commercial project, that can be fine. It’s all trade-offs. Faster velocity for slightly crappier code can be the different between profit and loss: actually shipping is of significant value.

But — and, again, this is my own view only — those trade offs apply above the foundation, above Django. There I need the known quality that I’ve always relied on.

To the extent that we allow Django’s coherence to erode, for its quality and maintainability to fall, we undermine the very reason to use it. Knock yourself out Claude, it’ll become “I need a web app” and not “I need a Django web app”, and at that point we can pack up shop.

Add ons? As I already said, I’m more receptive there: the splash-zone is contained.

As another example: I was talking with @jacobtylerwalls at Django on the Med about Django’s free-threading readiness. Having an LLM look for candidate problem areas, and stress test those, seems like a genuinely novel usage, that would have been difficult previously.

So, I don’t think plain refusal to engage is necessarily the right approach either.

The thing I’d like us to avoid is sleep walking into undermining the trust that people put in us. Ultimately, If it becomes a slop fest, that trust is gone. We’ll all just vibe up our own.

We have to have the conversation before that happens.

25 Likes

Thank you for starting this discussion - not quite sure where to start :slight_smile:

Firstly, we have the recent update to Django’s Code of Conduct which explicitly talks about AI usage: Django Code of Conduct | Django

Misuse of AI-generated content is a violation of our Code of Conduct.

Dan Ryan also did a talk about the Code of Conduct changes at DjangoCon US: https://youtu.be/kGhghG93CLw?si=yY-QIZVMTLYxpFNx
There was some discussion of AI here. Not sure if you have seen this talk yet but I found that really useful.

Secondly, on code attribution:

We remove co-authored by LLM attribution if it was added, as this has felt like free advertising for certain LLM models. It is also very hard for us to police this.
We do not use co-authored by for each PR reviewer and personally I want to credit reviewers before LLM models. This admin work would largely land on the fellows here. I am -1 on adding co-authored by for an LLM model.

Finally, our current code contribution model is that the community will review and approve contributions. Each code contribution has some form of review and approval from a merger.
Larger contributions require community discussion or a DEP.

Are you worried the mergers are merging slop? Or are you concerned that slop is thrown at the community to review?
I would like to think that the mergers and the community do a reasonable job with the quality gate, but if there are any commits with a quality issue, you (and anyone) can always raise a PR with a correction. We can even revert a commit.

I have been concerned about the amount that gets thrown at us but we are slowly doing steps to reduce this down. I am always open for new ideas here.

I am not against responsible use of AI. Especially to make the lives of the community easier, such as including a minimal test case or reproducer when you open a Trac ticket.
I am even open for us to have automatic security reviews of PRs, perhaps triggered by a community member. The security team needs to address anything that gets picked up from an AI scanner anyway, so I think better to resolve it before it goes in.


I wasn’t quite sure the direction you wanted this discussion to go based off your post.
So apologies if I am on a tangent to what you are after here :heart:

1 Like

Hey @sarahboyce,

Yes, so we (you!) manage to do a remarkable job given the influx. I was just looking at the Patches needing review metric, and how it’s been flying up since August (in particular):

That’s essentially 100 increase in that time. And I suspect most of them have hardly got a human behind them.

Is that the kind of contribution we want, I guess is part of my worry.

And then there is a further one, about the erosion of quality as machine written patches do get merged, even with folks trying to do their best.

(Just on the attribution question, there’s a lot of “I wrote…”, “I did…”, “I made…” when the claimant essentially wrote/did/made no such thing. It’s not that we should credit the models but that we’re not passing off machine work as human in origin. Don’t have an immediate suggestion, but it seems a problem.)

2 Likes

Thanks for bringing this. Actually i agree with Winset where he said “we should use LLM only when we can do that work manually” on one of the django chat episode. For me it make sense.

For example, if I’m submitting pr with help of LLM on project where I have no experience. Even if I used top tier model it will create slop anyhow.

I can see the similar slops in some new contributors PRs. The changes are too verbose, over engineered, irrelevant.

1 Like

Even if we decided this was an amazing idea, I think it would only have negative effects.

With prose, it’s relatively easy to detect LLM-authorship. With code (unless there are a lot of comments), this doesn’t really seem possible to me.

Without any enforcement mechanism you are reduced to guessing, and as models get better, as they continue to do, your guesses will become less and less accurate, and is probably not something you want people to spend their time doing anyway.

So there will be no decrease of bad actors spamming low effort PRs.

In my view, the people you hurt are good actors, who use LLMs thoughtfully to do work that they otherwise might not have time for but are of high quality and wanted by the project.

So personally I would much rather (somehow) increase the burden for the bad actors while encouraging those who do use LLMs thoughtfully and with skill, and making life easier for them, perhaps even by adding AGENTS.md or similar files in the right places so that coding agents can do the right things already without needing the user to prompt them to do it.

At the moment however I would think it prudent to try to enforce not using LLMs to write prose, especially documentation, and this is something that could likely be automated in some way, since LLM prose is still (for now) relatively easy to detect and usually obviously inferior to anything a human would write.

4 Likes

I think there is a very real danger that we are drifting into a position where we incrementally undermine the quality and stability that constitutes Django’s value as the framework of choice, without having had the proper discussion about how to maintain that quality, and what limits we might draw on machine generated code where that might undermine it.

I keep coming back to this paragraph, and I would like some clarification. Is there an example of this already happening, where machine-written code made it into Django and hurt its quality? Every release seems to build on twenty years of good releases, and I am super impressed by what Natalia, Sarah, and Jacob have put together for release notes and the release process.

When I first read your post, I was a little confused, because it felt like it was making points about structures we already have in place. We ask people to disclose AI use, our Code of Conduct covers misuse of AI-generated content, and big features still need a DEP that the Steering Council approves. I read the Django-Bolt and ORM examples as thought experiments, and I do not think AI changes anything there. If anything, it is amazing that people can now prototype a big feature quickly to see whether an idea works, even if we never accept that code.

I see disclosure differently. I agree that people should not pass off machine work as their own. But disclosure has completely changed over the last three to six months, to the point where we have to assume every contribution is LLM-driven. In the 2026 Django Developers Survey, 90% of respondents use AI weekly or daily. I do not know why new contributors would be any different.

Tom made some good points here. We cannot reliably tell when code was written by an LLM, so a rule against LLM code becomes guesswork, doesn’t stop bad actors, and hurts people who use LLMs thoughtfully. We cannot wish LLMs away, so I would rather make life easier for the contributors who use them well.

I brought up the patch queue today. Part of the growth is the shift in priorities toward 6.2, and Sarah is right that the volume is a lot. But if we are honest with ourselves, our process was stressed with one fellow, with two, and slightly less so with three. Between Trac and a CI setup where most builds were failing when I looked earlier today, it was not working, through no fault of anyone involved. AI only piled on. The camel’s back was already broken.

You say even the latest models are deeply flawed. My experience with current models is very different, and I think a lot of these threads are based on older ones. I think we should take a longer look in the mirror and admit what works and what does not. That could mean automating more for the fellows, and adding an AGENTS.md file so contributors’ agents start off on the right foot.

There still seems to be a disconnect between the tools our users use every day and the worry that using those same tools will lower Django’s quality. I have not seen evidence of that yet. If your point is that we want a more solid AI policy, then I think that is more than reasonable.

4 Likes

I feel good about Django’s quality, but our current contribution model invites bad-faith extractive contributions, and there is a big risk that we under-react.

I think we need a basis for saying, “you’re adding noise, go away”, but I also share the hesitations about equating mere AI usage to slop or noise.

On the queues and the extractive contributions:
For the slop that you can easily spot, I’ve done it all:

  • Polish (or rewrite) and merge, leaving the original author credit to avoid extra review
  • Close, assign to myself, rewrite, request a review
  • Deassign, throw back on the backlog
  • Ask questions, see if the author loses interest naturally

Option 1 is the only route that doesn’t elongate the backlog. I’ve chosen among those paths mostly out of expediency. For instance, the Security Team opens public tickets when we decline to confirm a CVE but something can be improved: Django gets a little more secure, and the Security Team gets fewer distracting duplicate reports. (I’m not thrilled with how this lets models dictate Django’s priorities, but that’s for another thread.) Since I’m very motivated to clear these tickets, I’ve probably over-indexed on option 1 (rewrite and merge without involving anyone else). This turns me into the operator of a candy machine. “Does your keyboard have an Enter key? Come on down to django/django and get an open source commit today!”

We need to have a basis for saying that trick-or-treat hours are over, while still welcoming new contributors. Sincerity is the only gate I can think of.

You can demonstrate sincerity by drafting a terrific bug report, by joining an in-person or virtual sprint or community event, by offering (non-LGTM) code reviews, by getting involved in the ecosystem, and so on. Djangonaut Space has a whole system to bring new folks up.

If that sounds like a huge gate, let’s add a backstop: in the absence of any obvious outward sincerity, reviewers can default to option 4: ask questions. If an author can respond in their own voice and can offer at least *one* non-sycophantic thought of their own, then we can keep your PR open. But that keeps PRs on the queue. (A Socratic seminar?)

Or, we don’t add a gate, and we just close or ignore noise arbitrarily. I closed some stuff today without comment, buoyed by this thread. The review queue (~200) reaching the asymptote of the number of accepted tickets (~1000) is something I really expected to have already concluded by now.

What I don’t think we can do is ask people point-blank if they’re a human, if they’re AI, if they’re sincere, if they know what they’re doing, what’s manage.py, etc. Insulting and unverifiable questions. AI disclosure is inherently in tension with AI bans.


On the slop you can’t spot, which is a major element in Carlton’s post. I don’t know what to do about it, but I can share my experience.

In my first weeks as a fellow, I related to @nessita at DjangoCon about chatting with a contributor at a sprint, then hearing myself agreeing to something that sounded reasonable, and then letting my guard down when checking the contribution. She reminded me to stay sharp and always assume things are wrong, even when checking each other’s work as fellows. She is right about that.

When I don’t have any basis to expect that a contributor knows anything at all about what they are doing, I have to multiply this attitude several times over. I don’t just assume a patch is wrong, I assume it’s solving the wrong problem in the wrong place at the wrong time and with the wrong testing strategy.


At the end of my day today I dropped in on @ontowhee’s regular live PR review session on Discord for the first time. She kindly agreed to look at something I was exhausted from reviewing. She advised me to reassign it to myself given the constant redirections and “sure thing boss, here you go”. I told her that we were reassigning it to ourselves by pairing on it in that very moment, and we could sort out the commit credits later. Before long we found a worthy question to ask. That’s the work. I’ll assign the commit credits however the community wants me to do it, but that’s the work, the rest is noise.

GitHub should probably display avatars of reviewers and testers on commits, not authors.

9 Likes

Yes. Exactly. :melting_face:

I find this hard to know even where to begin to engage with. Can you really not think that their output is flawed? Do you really not notice that every single time you use them?

My exact words were “the output of even the latest models is deeply flawed” (I added a bold). Unless you watch every step like a hawk, they quietly introduce entropy into your project, which is then taken as the foundation for the next step, and so then accumulates. If that’s not your experience, then you’re not looking. :person_shrugging:

Nope. The strongest models, across all the options. (And before the follow-up comes, yes, using it right too.)

I assume now we’re coming from such different starting points that agreement is likely impossible. (Can you really think that? :exploding_head:)

I think LLM output is (actually, but probably inevitably) just that little bit crapper than a human would create. (Assuming a baseline for the human, of course :melting_face:) — it’s that incremental worsening that is the danger over time:

That’s what I meant. I don’t know how else to say it.

But — given that I’m pretty sure you disagreed with all of that — even if we agree to pretend that the LLM output isn’t worse in any way — as if the latest model really did make the difference — merging it still makes Django harder to maintain. Historically patch authors are the key folks who we turn to when problems come up with the feature they added. But if those authors don’t understand that work, because they didn’t write it, then they can’t play that role.

“Oh, we can just set the LLM on it again, to fix whatever issue”.

Well, yes. And we can incrementally add that little bit more code that we don’t understand. (That we’re still imagining doesn’t hide subtle flaws.)

But that’s exactly Jacob’s “Does your keyboard have an Enter key?". That drives me straight back to Tim’s question: Why use Open Source at all? If we’re just proxying LLMs, what’s the point?

My hope was we’d have an (overdue) conversation:

So, yeah, that. (Again, I’m not how else to say it. :person_shrugging:)

That might be an outcome, yeah. I note the CoC clauses. They’re something, yes, but they’re quite slight. Do we really think that’s all we need to say?

Two threads have emerged:

  • My concern over maintaining the high standards we have in core — which things like the DEP process do help defend. But this whole thread was prompted by the — not unserious suggestion to just have Opus 5.5 write us API framework, so I don’t see those examples as purely hypothetical. (A written policy saying, “And, no, we won’t bypass the historical development process in order to do that kind of thing” would be more than we have now.)
  • The DoS on the open source project from a mass of drive-by LLM written PRs. It was always high volume. It was always hard. But it used to feel under pressure yet do-able. 526 PRs opened since 1 August (if I counted right) is something else entirely. (Maybe we should just set agents on them, and call it a day… :cocktail_glass:)

And just to note, again, I didn’t say we shouldn’t use 'em.

Carlton, I get that you are “not saying not to use them,” and you are saying that everything about them is deeply flawed. We can skip the quotes because I’m not agreeing or arguing. We can leave it at that.

  • My concern over maintaining the high standards we have in core — which things like the DEP process do help defend. But this whole thread was prompted by the — not unserious suggestion to just have Opus 5.5 write us API framework, so I don’t see those examples as purely hypothetical. (A written policy saying, “And, no, we won’t bypass the historical development process in order to do that kind of thing” would be more than we have now.)

I don’t see how this doesn’t already apply to the DEP process, but I trust you and the Steering Council to draft a policy/amendment, and it can be a quick win. I also trust the Fellows not to merge a rogue PR with a huge feature that’s not already approved/DEPed/followed the process.

  • The DoS on the open source project from a mass of drive-by LLM written PRs. It was always high volume. It was always hard. But it used to feel under pressure yet do-able. 526 PRs opened since 1 August (if I counted right) is something else entirely.

My only pushback is that the chart with open patches (not total open PRs, to be specific) didn’t paint the full picture. For anyone else reading/following, security reports were the biggest stress point, and we seem to have a much better handle on them than a year ago. See Jacob’s note about CVE vs. public tickets. If we should apply more resources to the open PR queue, then that can be a point of emphasis.

I don’t think the Steering Council can determine an AI policy without wider community consultation.

That’s why I wanted to open a discussion. An exploratory yes, and… phase is probably going to be more helpful than folks dividing into camps. Where’s the common ground?

It might be that I’m the only one at all concerned by these questions, but the reactions suggest I’m not.

Ok gotcha. It wasn’t clear that this was about an AI policy until you spelled it out.

As I mentioned, we at least need an AGENTS.md file (think MVP) to cover the gotchas we are commonly seeing. That shouldn’t require a DEP-level addition and might help dial down the annoyances until there is time for a more formal AI policy.

It might help everyone learn how the Fellows are using LLMs, what the Security team is doing, what other stakeholders are doing, and what is working versus what’s been a waste of time. Then we can learn more from the community. Sadly, we tried to kick off an AI Working Group, but it hasn’t borne any fruit yet; it’s on the board’s list to pick up and prioritize after the new year. When we formed it a year ago, we hoped it could help craft our AI policy.

Building on Jeff’s point about drafting a policy/amendment, I’d like to suggest a practical way to shape those rules using hard data rather than assumptions.

To bridge Carlton’s valid concerns about codebase entropy with the reality that AI tools are already here, I propose a two-step approach:

  1. A quick A/B test: Let’s take a small sample of recent PRs and review them in two parallel tracks one strictly human, and one AI-assisted. This will give us hard data on exactly where AI misses Django’s deeper architectural context.
  2. A data-driven AGENTS.md: We can use the blind spots identified in that test to write a comprehensive AGENTS.md file. Instead of an unenforceable ban, this acts as a “system prompt” to enforce Django’s architectural rules right inside the contributor’s IDE.

To help protect the maintainers’ and Fellows’ time, we could also include a baseline rule in that file: contributors must manually explain their architectural reasoning. If a contributor can’t explain the why, we don’t spend time reviewing the how.

This gives us a proactive, data-backed guide while the broader AI policy is formalized.

Thanks for opening this discussion Carlton. I’ve written a blog post to share my thoughts here: Django and LLMs | Lily's Blog . To summarise, I share your fears and I hope we find a way forward that keeps Django a place for human collaboration and community.

2 Likes

I’m assuming the risks here are Fellow and contributor burnout, but if there are others, please share.

If we’re concerned with emptying people’s wells because there are too many people asking for something from us, are there signals that we can look out for that are early warning signs? For the Fellows, maybe they should start reporting to their line supervisor a happiness/satisfaction metric. I get that that’s silly, but it does force a person to put a number to a feeling and allows both you and another person to see where that is over time. When that number goes below some threshold, time to do some extra recovery / adjust how you’re working. Goofy, maybe, but it’s better than waking up one morning and not wanting to get out of bed.

In general, perhaps we need to reset expectations around what’s expected/normal for this project. There’s nothing inherently unsustainable about having an open PR for every ticket. It’s code for discussion when someone is able to discuss a topic. As long as people have a mechanism of how to effectively triage, review and collaborate, then that code is just an artifact of the discussion.

One option is to define a minimum number/percent of first-time contributor PRs to engage with per month. That would give Fellows and community members some clarity around how to pick where to participate and set expectations for new contributors.

A second option is to continue our work on @ertgl’s PR for djangoproject.com to expand the member’s profile. This is the first step in allowing us to have a dashboard where we can see an author’s various community memberships next to their contribution.

A third option is to define a sub-role of the Triage and Review team (or a new team altogether), whose role is to review first-time contributor’s PRs first before having anyone else look at them. I see this similar to people who look at the new posts on reddit or hacker news. It’s also a reproduction of the security teams’ triager role. These folks would basically do the first round of communicating with people and sorting out whether a PR is something worth a deeper dive from the community. And again, combined with a reasonable goal on how many we review, there’s a clear expectation for first time contributors to get a response.

2 Likes

I think the risk I had in mind was investing more (and eventually, total) reviewer effort in ways that provide diminishing returns to the project. The impact is felt on the project, not on the maintainer.

The best case for open contribution models was that we all maintain this code together. Someone new comes along, and if they have a positive experience, they come back, and over time they build knowledge of code, architecture, decision history, etc., and maybe even join up as maintainers.

If current trends continue, we are in stark danger of investing all of our available maintenance cycles talking to only lightly proxied LLMs.

For some of these extractive contributions, it appears that having a purple “Merged” icon on a pull request is more important than arriving at the best decision for the project. I get the feeling that if I asked the author to insert divide-by-zero errors, they would happily do it. (“Sure thing boss, here you go.”) It’s hard to ignore the feeling that investing in PRs by lightly proxied LLMs is a bad investment of time. This is why I discussed sincerity above.

A contributor told me recently, “when I can tell it’s an agent, I just unsubscribe. But you’re a Fellow, so I know you can’t do that.” I think Carlton is asking if we all can, in fact, unsubscribe.

So this isn’t about maintainer burnout or mental health. My blood pressure won’t rise if the open pull requests go above a certain big round number. (Or if it does, I’ll unplug and get help.) It’s about expectations setting.

I’m glad we’re discussing this. I ventured this same thought to someone at DjangoCon Europe in April – is the exploding backlog just an optics problem? – and I could barely get to the end of my sentence before they exclaimed, “no, it’s a huge problem!”

I suppose the huge problem is that everyone waits longer for a review, both the sincere new person who would like to learn, as well as the experienced contributor trying to chip away at a multi-step roadmap.

If we all pivot to this attitude, that could work just fine. But I think it’s very different from our default attitude, where we are accustomed to helping bring an author’s original effort to fruition. Nobody likes telling someone “sorry about all the time you invested, but we need to start over.” Tim, I very much like your blog post about how to do code reviews. In it, you caution “are you trying to make the other developer write your code?”:

If you’ve delegated some work to another person they are going to do it as they feel is best. … If it’s a small thing and doesn’t have cascading effects, I like to acknowledge the solution and let them know that it’s good to go, then bring up the way I would have approached it.

This works great for humans, but if the other party is an agent that’s going to be suspended by GitHub in two weeks for spamming open source projects, there’s no value in imparting any alternative approaches, and thus no value in trying to bring their original effort to fruition. They won’t be around when we need them. No one has effectively “delegated” anything to an agent. They just showed up. I fear the discussions won’t keep this in mind, and will default toward being agreeable. (Carlton’s watch them like a hawk above; Natalia’s (paraphrased) always assume that it is wrong.)

It also changes the tenor of the pull requests tab if we turn it into a “discussions” tab. (I don’t think of it as a discussions tab; my perspective is: if a PR is open, the next step is to advance it by pushing to it.)

In the absence of any other tools to gauge sincerity, I guess I’d rather see us define a maximum. Over half of the pulls on the first page of pull requests at this moment are from new contributors. The most recent is from a first-time GitHub contributor. There’s no shortage of new accounts to iterate with. It can consume all of the available oxygen. I have to make a special effort to seek out existing contributors to iterate with.

Thanks for bringing this to my attention! Using this to slice and dice the open pull requests sounds intriguing.

I’m down to try anything. But I fear that there is no quick win here. Most LLM output looks reasonable on first glance, so I’m not sure how the light screen would play out.

2 Likes