Using a Gmail account to send email

I have been using Gmail to send emails with Django for a few years now, but I am confused about how I am supposed to authenticate with Gmail because I have twice received emails (not directly, details below) from Google saying that our authentication method was being deprecated.

First email from Google in 2019: " Starting February 15, 2021, G Suite accounts will only allow access to apps using OAuth. Password-based access will no longer be supported. "

So I moved to using GitHub - dolfim/django-gmailapi-backend: Email backend for Django which sends email via the Gmail API to authenticate with OAuth, but I have recently received an email that they are deprecating out of band tokens, which that library uses: Google Developers Blog: Making Google OAuth interactions safer by using more secure OAuth flows . I donā€™t see any clear way to authenticate an app to send emails in their recommended migrations. They almost all seem centered around approving permissions for users rather than the app itself. I have opened an issue on the library OAuth out-of-band token deprecation Ā· Issue #11 Ā· dolfim/django-gmailapi-backend Ā· GitHub, but there wasnā€™t a consensus on how to handle this.

One option was to use an app password, but that sounds like what I was doing initially. The other option I thought of was to create a page that only I can access to authenticate the application with OAuth, but that seems like a roundabout way of handling this.

I have reached out to Googleā€™s Workspace team about this and should be talking to them soon, but in the few email exchanges Iā€™ve had, they didnā€™t seem to understand what I was requesting. Iā€™ll update if they give me any relevant information.

I am considering moving to SendGrid or some other service that handles email, but that feels unnecessary since we send so few emails.

Is there some easier way to authenticate with Gmail that Iā€™m missing?

Note: My interpretation is that Gmail is eliminating that mode of authentication for their web interfaces. I have not seen anything stating that SMTP authentication is changing.

(I donā€™t use the gmail API, I send email through them using SMTP - works great, hasnā€™t given me any problems.)

1 Like

Thatā€™s what I was doing initially using a password (not the account password, but a generated password through the control panel; I forget what it was officially called). Then I got that first email. Are you doing something different that meant you didnā€™t get that email?

What method are you using to authenticate using SMTP?

This is what my settings.py looked like before switching to OAuth:

EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp-relay.gmail.com"
EMAIL_HOST_PASSWORD = "password"
EMAIL_HOST_USER = "address@gmail.com"

No, I got the email - but after digging through what I could find of their documentation, I came to the (quite possibly erroneous) conclusion that what theyā€™re doing doesnā€™t affect SMTP authentication.

Yes, your settings fundamentally look like what Iā€™m doing.

(Except that I donā€™t send emails directly from Django. I run a local Postfix server that is the target of my applications, and it forwards the emails out to gmail.com.)

2 Likes

Great, thanks for the info. I was planning to try the app password again in any case, but this makes me feel like itā€™s probably the way to go. Iā€™ll update if I hear anything of value of from the Google Workspace team when I finally talk to them.

1 Like

I spoke to a support person with Google Workspace, and he told me that app passwords still work because too many folks were using them for Google to shut down that method of authentication. They still consider it unsafe and they would like to deprecate it, but they donā€™t have a timeline yet.

He sent me some information about how to possibly authenticate using impersonation in order to continue using the aforementioned library, but Iā€™m not convinced it is an easy fix. Unfortunately, it doesnā€™t seem as though Google has an easy way to solve this problem.

Iā€™m going to try moving back to the app password, and if it does get deprecated in the future, Iā€™ll probably move to a paid service.

Sorry to revive this old thread, but Iā€™m coming here after finding out that the deprecation is coming Autumn 2024.
It specifically names SMTP:

CalDAV, CardDAV, IMAP, SMTP, and POP will no longer work with legacy passwords (basic authentication).

Has django-gmailapi-backend fixed the out-of-band tokens? Is there another solution that has developed in the past year?

Thanks for pointing this out.

However, Iā€™m only seeing this being referenced in terms of Google Workspace accounts, not personal GMail accounts.

Have you found any reference for this regarding personal GMail accounts? (Iā€™ve been searching since you posted this and havenā€™t yet found anything.)

1 Like

Less secure apps & your Google Account - Google Account Help says that personal Gmail accounts should have already been forced over, so maybe this is only for your account password and not for generated app passwords? Iā€™m confused, but thatā€™s not unusual when it comes to this topic and Google.

I have gone back to using an app password and havenā€™t had any issues. Also, last time they threatened this, they sent quite a few emails in advance, and I havenā€™t received anything yet.

Alright, Iā€™ll continue using a Google App Password configuration in my settings.py.
Weā€™ll see if it continues working by the end of next year.

EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.gmail.com"
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_HOST_USER = "emailCreatedOnlyForApp@myDomain.com"
# This is a Google App Password from https://myaccount.google.com/u/0/apppasswords
# Be sure to log in to that link with the above email address
EMAIL_HOST_PASSWORD = "abcdefghijklmnop"

it seems like this will no longer work after june 2024 as is. see:

you will have to secure the email user with 0auth and then create an app pasword for sending emails through smtp. correct me if i am wrong. thanks!

Disclaimer: We here donā€™t have any authoritative information. If youā€™re looking for definitive answers, youā€™ll need to get your questions answered by Google. All the information provided here on this topic represents our understanding of the information being provided.

Having said thatā€¦

Here, I donā€™t think youā€™re right. Quoting from that post:

ā€¦ youā€™ll need to either: configure them to use OAuth, use an alternative method, or configure an App Password for use with the device.

I read this as saying you have three different options here, not that you need to ā€œuse Oauth and create an app passwordā€.

Additionally, I still only see the emphasis here on Google Workspace accounts, not the typical individual gmail account.

Most importantly, Iā€™m still not finding anything saying that the app password facility is going away.

thanks for the reply. and yes, App Password is still available, but the email account has to have 2FA enabled. if you are sending emails from many different email accounts, this becomes a bit of problem. for my part, we are migrating to using the gmail API for sending emails.

@bradleo99 ā€¦ Google disabled username/password SMTP authentication for my Workspace already. Unless they were talking about the Southern Hemisphere autumn when they said ā€œFall 2024ā€ theyā€™re 6 months early. Itā€™s been an interesting day scrambling to scrape the required info out of the labyrinthine Gcloud docs. Donā€™t rely on Gemini for anything sensible - it prefers quoting 12 year old StackOverflow posts to its own documentation ā€¦ or just fabricating nonsense. My swear jar is now a swear bucket.

The out-of-bounds issue with django-gmailapi-backend isnā€™t fixed - I tried this with a new gcloud project and Google said no, returned a ā€œnot allowedā€ error.

I ended up adding a service account to my project, adding the necessary ā€˜send asā€™ delegation in the Workspace admin, then sending email using this service account.

The delegation needs the following scopes:
https://www.googleapis.com/auth/gmail.send https://www.googleapis.com/auth/gmail.compose

I altered the send_message.py found on the docs to arrive at:

import base64
import logging
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError

from site_settings.models import Tokens


def gmail_send_message(sender, recipients, subject, content, cc=None, bcc=None):
    try:
        credentials = service_account.Credentials.from_service_account_info(
            Tokens.load().gmail_service_account, 
            scopes=['https://www.googleapis.com/auth/gmail.send']
        ).with_subject(sender)

        service = build("gmail", "v1", credentials=credentials)

        # Create a multipart message container
        message = MIMEMultipart("alternative")
        message["To"] = ", ".join(recipients) if isinstance(recipients, list) else recipients
        if cc: message["Cc"] = ", ".join(cc) if isinstance(cc, list) else cc
        if bcc: message["Bcc"] = ", ".join(bcc) if isinstance(bcc, list) else bcc
        message["From"] = sender
        message["Subject"] = subject

        # Create the HTML part of the message
        html_content = MIMEText(content, "html")

        # Attach the HTML part to the message
        message.attach(html_content)

        # Encode the message as base64
        create_message = {
            "raw": base64.urlsafe_b64encode(message.as_bytes()).decode()
        }

        # Send the message
        send_message = (
            service.users()
            .messages()
            .send(userId="me", body=create_message)
            .execute()
        )
    except HttpError as e:
        logging.error(
            f"{type(e).__name__} at line {e.__traceback__.tb_lineno} of {__file__}: {e}"
        )
        send_message = None
    return send_message

The amended version uses the service account to send of behalf of the address in sender.
Supply recipients (required), cc (optional) and bcc (optional) as a single address of list of addresses. content can be plain text or html.

  • Tokens.gmail_service_account is a JSONField in my custom Tokens model that holds the contents of the service account key file. You can also use service_account.Credentials.from_service_account_file() to read the file directly if you prefer (I donā€™t like token files sitting around the server myself).

  • When youā€™re creating the credentials, you must add .with_subject(sender) where sender is the email address youā€™re sending on behalf of (and this address must have granted delegation to the service account). Without this, the send will fail.

Maybe this gives you something to work off. Definitely a ā€˜heads upā€™ that Google arenā€™t honouring their published timeframe.

Thanks for the heads up. As django-gmailapi-backend hasnā€™t had any commits in 2 years (and barely has any github stars to begin with) I will assume it will never be updated. When Google decides to cut my Workspace over I will absolutely use your code as a springboard!

Donā€™t expect any forewarning from Google, they cut mine off without notice ā€¦

Once the dust had settled and Iā€™d worked out how to send native django emails over the Gmail API, this became a much easier task as I could just clone the default SMTP Django backend and adjust. Means all those built-in apps using sendmail() and message.send() still work without needing to change anything there. The gmail_send_message() method is redundant now.

# core/mail/backends.py
import base64
import copy
import logging
import threading

from django.conf import settings
from django.core.mail.backends.base import BaseEmailBackend
from django.core.mail.message import sanitize_address
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError

class GmailBackend(BaseEmailBackend):
    """
    Django email backend for sending with Google Workspace
    """
    def __init__(self, fail_silently=False, **kwargs):
        self.fail_silently = fail_silently
        self._lock = threading.RLock()
        service_account_key = settings.GMAIL_SERVICE_KEY
        # base_credentials used to build user specific credentials in send method
        self.base_credentials = service_account.Credentials.from_service_account_info(
            service_account_key,
            scopes=['https://www.googleapis.com/auth/gmail.send']
        )

    def open(self):
        """ retained for legacy code compaitibility """
        return True
    
    def close(self):
        """ retained for legacy code compaitibility """
        pass

    def send_messages(self, email_messages, thread=False):
        """
        Sends one or more EmailMessage objects and returns the number of email
        messages sent.
        """
        if not email_messages:
            return 0
        with self._lock:
            num_sent = 0
            for message in email_messages:
                sent = self._send(message)
                if sent:
                    num_sent += 1
        return num_sent

    def _send(self, email_message):
        """
        A helper method that does the actual sending.
        """
        if not email_message.recipients():
            return False
        try:
            # sanitize addresses
            sanitized_message = self.sanitize_addresses(email_message)
            message = sanitized_message.message()
            # Gmail api send() requires JSON format base64 string - convert .message() before sending
            raw = {"raw": base64.urlsafe_b64encode(message.as_bytes(linesep="\r\n")).decode("utf-8")}
            # Set delegated credentials according to the sender email address
            delegated_credentials = self.base_credentials.with_subject(email_message.from_email)
            service = build("gmail", "v1", credentials=delegated_credentials )
            # Send email
            service.users().messages().send(userId='me', body=raw).execute()
        except HttpError as e:
            logging.error(
                f"{type(e).__name__} at line {e.__traceback__.tb_lineno} of {__file__}: {e}"
            )
            if not self.fail_silently:
                raise
            return False
        return True
    
    def sanitize_addresses(self, email_message):
        """ 
        Make a copy of email message (email_message passed by reference), sanitize addresses 
        Ensure that email addresses are properly formatted & without potentially harmful characters
        """
        message_copy = copy.copy(email_message)
        encoding = email_message.encoding or getattr(settings, 'DEFAULT_CHARSET', 'utf-8')
        message_copy.from_email = sanitize_address(email_message.from_email, encoding)
        for field in ['to', 'cc', 'bcc', 'reply_to']:
            setattr(message_copy, field, [sanitize_address(addr, encoding) for addr in getattr(email_message, field)])
        return message_copy

The local settings just need the following to configure mail:

EMAIL_BACKEND = "core.mail.backends.GmailBackend"
DEFAULT_FROM_EMAIL = EMAIL_HOST_USER = 'someone@somewhere.com'
GMAIL_SERVICE_KEY = { ... }

The delegation is all or nothing unfortunately - you canā€™t grant delegation rights to only certain mailboxes, it has to be domain wide. I tried granting send-on-behalf rights to the service account in the gmail client, computer said no, the service account isnā€™t recognised as a Google account. Nice consistency there, other Google apps will let you do this, just not Gmail. You can add some logic to the backend to restrict ā€˜fromā€™ addresses but if someone gets access to the key then thereā€™s no restriction. The alternative is working out how to generate the refresh tokens reliably and modifying the backend to use that instead.

Possible am only left with using mailgun or any other.coz surely i have failed