Create users from excel file

ArianN8610 · October 23, 2024, 5:52pm

I’m trying to create a structure where I can create all users with full profile through an excel file without having to manually enter them:
view:

class ImportUserView(PermissionRequiredMixin, FormView):
    permission_required = 'account.add_user'
    template_name = 'dashboard/import_users.html'
    form_class = forms.ImportUserForm
    success_url = '/dashboard/users/'

    def form_valid(self, form):
        excel_file = form.cleaned_data['user_file']
        index_excel = 0

        try:
            with transaction.atomic():
                wb = load_workbook(excel_file)
                sheet = wb.active

                for index, row in enumerate(sheet.iter_rows(min_row=1)):
                    index_excel = index + 1

                    # Create records in db
                    user = models.User(username=row[0].value)
                    user.set_password(str(row[1].value))
                    user.full_clean()
                    user.save()

            messages.success(self.request, 'success')
        except Exception as e:
            form.add_error(None, f'error: line {index_excel}')
            form.add_error(None, e)
            return self.form_invalid(form)

        return super().form_valid(form)

form:

class ImportUserForm(forms.Form):
    user_file = forms.FileField(
        label='Users',
        widget=forms.FileInput(attrs={
            'accept': '.xlsx'
        }))

I run the project with docker and nginx. docker-compose:

services:
  backend:
    build: .
    container_name: backend
    command: gunicorn main.wsgi:application --bind 0.0.0.0:8000
    privileged: true
    restart: always
    volumes:
      - ./core/:/home/app/web/
      - ./core/static:/home/app/web/static
      - ./core/media:/home/app/web/media
    expose:
      - "8000"
    env_file:
      - ./.env.prod
    depends_on:
      - db

  db:
    image: postgres:15-alpine
    restart: always
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./.env.prod.db

  nginx:
    build: ./nginx
    restart: always
    privileged: true
    ports:
      - "80:80"
    volumes:
      - ./core/static:/home/app/web/static
      - ./core/media:/home/app/web/media
    depends_on:
      - backend


volumes:
  static_volume:
  media_volume:
  postgres_data:

nginx-config:

upstream shb {
    server backend:8000;
}

server {

    listen 80;
    client_max_body_size 64M;

    location /static/ {
      alias /home/app/web/static/;
    }

    location /media/ {
      alias /home/app/web/media/;
    }

    location / {
        proxy_pass http://shb;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

}

The program works without problems on the local system and adds users. On the local system, everything is similar to prod, with one difference in docker-compose:

services:
  backend:
    build: .
    container_name: backend
    command: python manage.py runserver 0.0.0.0:8000
    privileged: true
    restart: always
    volumes:
      - ./core/:/home/app/web/
      - ./core/staticfiles:/home/app/web/staticfiles
      - ./core/media:/home/app/web/media
    ports:
      - "8000:8000"
    env_file:
      - ./.env.dev
    depends_on:
      - db

  db:
    image: postgres:15-alpine
    restart: always
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./.env.prod.db


volumes:
  static_volume:
  media_volume:
  postgres_data:

I don’t use nginx in development
But in prod, after uploading the file and adding some values, I see the “Internal Server Error” error.
backend container logs:

backend  | Traceback (most recent call last):
backend  |   File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
backend  |     self.handle_request(listener, req, client, addr)
backend  |   File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
backend  |     respiter = self.wsgi(environ, resp.start_response)
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
backend  |     response = self.get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 140, in get_response
backend  |     response = self._middleware_chain(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/deprecation.py", line 134, in __call__
backend  |     response = response or self.get_response(request)
backend  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django_htmx/middleware.py", line 43, in __call__
backend  |     return self.get_response(request)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/home/app/web/account/middleware.py", line 17, in __call__
backend  |     response = self.get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
backend  |     response = get_response(request)
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
backend  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 104, in view
backend  |     return self.dispatch(request, *args, **kwargs)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
backend  |     return super().dispatch(request, *args, **kwargs)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 143, in dispatch
backend  |     return handler(request, *args, **kwargs)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/views/generic/edit.py", line 153, in post
backend  |     return self.form_valid(form)
backend  |            ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/home/app/web/dashboard/views.py", line 164, in form_valid
backend  |     user.set_password(str(row[1].value).strip())  # Save password for user
backend  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/contrib/auth/base_user.py", line 108, in set_password
backend  |     self.password = make_password(raw_password)
backend  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/contrib/auth/hashers.py", line 97, in make_password
backend  |     return hasher.encode(password, salt)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/contrib/auth/hashers.py", line 311, in encode
backend  |     hash = pbkdf2(password, salt, iterations, digest=self.digest)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/django/utils/crypto.py", line 77, in pbkdf2
backend  |     return hashlib.pbkdf2_hmac(digest().name, password, salt, iterations, dklen)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
backend  |     sys.exit(1)
backend  | SystemExit: 1

Apparently, the problem is the password storage for one of the users. But I checked the password of that user and there was no particular problem and it was like the others. I also use str() so that the value must be given as str to set_password. what is the problem

KenWhitesell · October 23, 2024, 10:48pm

I would suggest adding a logging statement to log the raw password. I might even suggest logging the hex representation of the string to ensure there are no non-visible characters embedded that you wouldn’t see with a visual inspection.

ArianN8610 · October 24, 2024, 6:42am

I checked the password that caused the error, there was no problem, all its characters are numbers and I put it in the str function, so there should be no problem. It also works correctly on my local system and the only difference between the server and the local system is nginx. I think maybe the problem is with my nginx settings, but I’m not sure
And I write this part of my code in try, but when an error occurs, why does it not work and I see “Internal Server Error”?

boxed · October 24, 2024, 7:35am

This looks like the worker getting killed because it’s exceeding the timeout to me. Does this processing take a long time?

ArianN8610 · October 24, 2024, 8:43am

Yes, it does. because the number of users is large. It creates about 37 users, but it gives this error after 37 users

leandrodesouzadev · October 24, 2024, 11:04am

It looks like you’re indeed receiving a worker timeout error, from gunicorn. From this line you can tell.

If you’re not explicitly defining a timeout gunicorn defaults the timeout to 30 seconds, docs.
If this really takes a long time, you should consider either upgrading the timeout setting as a duct tape fix. But take in consideration that having a long timeout for your workers is not a good practice. You may also end up stumbling across nginx timeout that is 60 seconds by default.

You should consider doing this the right way, that is processing it on the background with a tool like celery that is designed for operations that take a long time.

Topic		Replies	Views
Dúvidas na geração de arquivos Excel. Using Django	2	536	January 23, 2021
ProgrammingError while migrating Django DB to Postgre Using Django	19	4955	August 12, 2020
login issue for imported users Using Django	1	502	January 31, 2022
Site fails (Database error) when deployed from local (SQLite) to heroku (PostgreSQL) Using Django	0	1115	January 10, 2022
How to bulk import users Using Django	13	3545	November 5, 2021

Create users from excel file

Related topics