ConnectionResetError(104, 'Connection reset by peer')

Using Django
1

I have an app deployed on Google App Engine which every couple of days is sending me an error email:

Internal Server Error: /

ConnectionError at /
('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I am aware that this has been discussed in various places before, but the discussed solutions are so varied, and this error is so sporadic, that it is becoming difficult to resolve.

Django version is 3.1.4 and Python 3.7.9.

Here are a few things I’ve noticed:

  • It only seems to occur on the root page (‘/’).
  • The user agent seems to be a web crawler.
  • I can’t trigger the error myself, nor have I seen another real user trigger it.
  • The stack trace is always:

Traceback (most recent call last):
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 677, in urlopen
chunked=chunked,
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 381, in _make_request
self._validate_conn(conn)
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 978, in validate_conn
conn.connect()
File “/env/lib/python3.7/site-packages/urllib3/connection.py”, line 371, in connect
ssl_context=context,
File "/env/lib/python3.7/site-packages/urllib3/util/ssl.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File “/opt/python3.7/lib/python3.7/ssl.py”, line 423, in wrap_socket
session=session
File “/opt/python3.7/lib/python3.7/ssl.py”, line 870, in _create
self.do_handshake()
File “/opt/python3.7/lib/python3.7/ssl.py”, line 1139, in do_handshake
self._sslobj.do_handshake()

During handling of the above exception ([Errno 104] Connection reset by peer), another exception occurred:
File “/env/lib/python3.7/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “/env/lib/python3.7/site-packages/urllib3/util/retry.py”, line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/env/lib/python3.7/site-packages/urllib3/packages/six.py”, line 734, in reraise
raise value.with_traceback(tb)
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 677, in urlopen
chunked=chunked,
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 381, in _make_request
self._validate_conn(conn)
File “/env/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 978, in validate_conn
conn.connect()
File “/env/lib/python3.7/site-packages/urllib3/connection.py”, line 371, in connect
ssl_context=context,
File "/env/lib/python3.7/site-packages/urllib3/util/ssl.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File “/opt/python3.7/lib/python3.7/ssl.py”, line 423, in wrap_socket
session=session
File “/opt/python3.7/lib/python3.7/ssl.py”, line 870, in _create
self.do_handshake()
File “/opt/python3.7/lib/python3.7/ssl.py”, line 1139, in do_handshake
self._sslobj.do_handshake()

During handling of the above exception ((‘Connection aborted.’, ConnectionResetError(104, ‘Connection reset by peer’))), another exception occurred:
File “/env/lib/python3.7/site-packages/django/core/handlers/exception.py”, line 47, in inner
response = get_response(request)
File “/env/lib/python3.7/site-packages/django/core/handlers/base.py”, line 179, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/opt/python3.7/lib/python3.7/contextlib.py”, line 74, in inner
return func(*args, **kwds)
File “/home/vmagent/app/shop/views.py”, line 48, in index
tweets = twitter_client.timeline(requested_timeline)
File “/home/vmagent/app/shop/twitter.py”, line 37, in timeline
user_id = self.get_user(user)
File “/home/vmagent/app/shop/twitter.py”, line 31, in get_user
response = requests.get(url, headers=self.headers)
File “/env/lib/python3.7/site-packages/requests/api.py”, line 76, in get
return request(‘get’, url, params=params, **kwargs)
File “/env/lib/python3.7/site-packages/requests/api.py”, line 61, in request
return session.request(method=method, url=url, **kwargs)
File “/env/lib/python3.7/site-packages/requests/sessions.py”, line 530, in request
resp = self.send(prep, **send_kwargs)
File “/env/lib/python3.7/site-packages/requests/sessions.py”, line 643, in send
r = adapter.send(request, **kwargs)
File “/env/lib/python3.7/site-packages/requests/adapters.py”, line 498, in send
raise ConnectionError(err, request=request)

I’m thinking that maybe something is reloading on the server when a web crawler visits the page. Is there anything I should take a closer look at or can I filter this error?

2

Very superficially, it looks like the type of error you might get when you try to connect to a server that is expecting an https connection without ssl, or the reverse (server is expecting an unencrypted connection and the client is trying to use https.)

This would be happening at a very low layer - I’m not sure if it can be handled at the application layer.

3

In the original exception’s stack trace, I can’t see any application logic. To me, it appears to be code used to make a request to another server and not typical application processing. Unfortunately, without more information on how that flow is started and which part of your application code does that, it’s going to be difficult to debug.

That said, the last exception in your post caught my eye. See highlighted section below:

You said that you were able to determine the caller is a crawler of sorts. Was it requesting data with valid parameters? Ideally you’d be able to see the string representation of the locals for each step of the stack traces. You can do that by integrated an error monitoring solution like Sentry, or print the relevant parts.

My guess here is that the request is actually invalid and when you attempt to fetch the user info from Twitter things are breaking because of Twitter’s rule. This could align with Ken’s response if the url in response = requests.get(url, headers=self.headers) isn’t using https.

Finally, I’m not sure which part of the twitter API you’re using, but if it’s the streaming timeline api, there’s a mention of handling disconnects in step 4 of the tutorial.

4

Thanks both.

The app does perform an API call to v2 of the Twitter API.

In terms of the protocol, the view is accessed via https and the Twitter endpoint also uses https. So there shouldn’t be any possibility of a http/https conflict.

All of the parameters necessary to perform the Twitter request are provided automatically, so there shouldn’t be an issue with the user providing invalid data.

I also noticed at least one of the requests came from China, where I believe Twitter is blocked (correct me if wrong). That might be the underlying issue if I can’t reproduce this error here. Twitter could be blocking whatever web crawler is trying to perform the request based on the location.

I think wrapping the Twitter request in a try/except block and handling the ConnectionError is potentially the most straightforward option. It will take a while to see if that works, but hopefully that makes sense?