New error: message handler error

[MathiasDrapier]

Hello,

I have had a socketio flask server running well for several months.
For a few days I observe an error that I had never had before.
Knowing that I did not modify my code.

Here are the logs displayed in my docker container:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 824, in _handle_event
    ret = handler(auth)
TypeError: handle_connect() takes 0 positional arguments but 1 was given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 679, in _handle_connect
    success = self._trigger_event(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 826, in _handle_event
    ret = handler()
  File "/usr/src/app/src/socketManager.py", line 32, in handle_connect
    join_room(room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 1035, in join_room
    socketio.server.enter_room(sid, room, namespace=namespace)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 436, in enter_room
    self.manager.enter_room(sid, namespace, room)
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 125, in enter_room
    eio_sid = self.rooms[namespace][None][sid]
  File "/usr/local/lib/python3.9/site-packages/bidict/_base.py", line 523, in __getitem__
    return self._fwdm[key]
KeyError: 'vG3PNlRagRVOmxYsAq79'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'VCKrq0rqQLwQRGajAxVl'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'endGTeqtyOeCdPWCA0Ke'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: '5K4MYuiP1jHyDcsdA0Kc'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'aiIHD8EB-dR0JYerA0UU'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'K1DZtsK79DdUUL4zA0Vq'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'mk0HyOgKl5NQ3RKiA0Vr'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: '2ND8onb7p11x55SxA0Vp'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'LXlrOxov8rRodyomA0Vt'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'zqxpfT-rf_DckWeWA0Wk'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: '5er3YoJOwZU-Ci5vA0Wn'
message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'mqO25BmyfLtkFEXrA0Wj'
message handler error

I did not put everything, but I have a repetition of lines:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: 'zqxpfT-rf_DckWeWA0Wk'
message handler error

, with different keyError each time.

Here is the code for my socketManager:

@socketio.on('connect')
def handle_connect():
    token = getToken(request)
    if token is not None:
        if ensureToken(token, conn[1]):
			room = str(getId(token, conn[1]))
			join_room(room)
		else:
			return False	
    else:
        return False
		
@socketio.on('message')
def handle_message(data):
    roomsList = rooms()
    for room in roomsList:
        emit('message', data, to=room, skip_sid=request.sid)

@socketio.on('disconnect')
def handle_disconnect():
    roomList = rooms()
    for room in roomList:
        leave_room(room)

[miguelgrinberg]

I'm not completely sure of the cause of these errors. The issue is that your server is receiving requests that have invalid sid values that are unknown, or maybe they were valid at some point but the client then disconnected and is now trying to communicate using these stale sids.

Are you doing anything out of the ordinary in the client side? Which client are you using?

[MathiasDrapier]

Hello thanks for your answer,
No I'm not doing anything unusual and I haven't changed anything on my client. This problem has appeared for a few days while no modification has been made in the code.
The problem isn't actually this error, it's that it crashes my server and after that nothing works (http and socketio). I have to restart it. Is there a solution so that this error does not cause a crash?

[miguelgrinberg]

Well, none of the logs that you share show a crash. These are just stack traces that were logged, there is nothing in these errors that suggests the server stops running. If you have a way to reproduce this error easily you can enable the Socket.IO logs and capture a session of logs until the failure, plus logs for the clients, and then maybe there will be more clues.

[MathiasDrapier]

No, I don't know how to reproduce this bug. And after these logs my server no longer responds (http and socketio) I don't understand why

[MathiasDrapier]

Hello, I allow myself to rewrite in this discussion because my problem is not solved and it disrupts the operation of my server enormously. As I said previously, once the socketIo server crashes, nothing works on my server (no more http request response from flask and no more socketio).

I enabled the logs and saved them in a txt file and waited for the bugs to occur. I get a 3giga file, with always the same sequence of logs:

2023-03-04 03:17:51,002 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 790, in _handle_event
    environ = self.server.get_environ(sid, namespace=namespace)
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1519, in findCaller
    f = currentframe()
  File "/usr/local/lib/python3.9/logging/__init__.py", line 160, in <lambda>
    currentframe = lambda: sys._getframe(3)
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1187, in emit
    StreamHandler.emit(self, record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 671, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 621, in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
  File "/usr/local/lib/python3.9/traceback.py", line 103, in print_exception
    for line in TracebackException(
  File "/usr/local/lib/python3.9/traceback.py", line 517, in __init__
    self.stack = StackSummary.extract(
  File "/usr/local/lib/python3.9/traceback.py", line 366, in extract
    f.line
  File "/usr/local/lib/python3.9/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/local/lib/python3.9/linecache.py", line 30, in getline
    lines = getlines(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 46, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/local/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
RecursionError: maximum recursion depth exceeded while calling a Python object
2023-03-04 03:17:51,007 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

During handling of the above exception, another exception occurred:

RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1187, in emit
    StreamHandler.emit(self, record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 671, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 621, in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
  File "/usr/local/lib/python3.9/traceback.py", line 103, in print_exception
    for line in TracebackException(
  File "/usr/local/lib/python3.9/traceback.py", line 517, in __init__
    self.stack = StackSummary.extract(
  File "/usr/local/lib/python3.9/traceback.py", line 366, in extract
    f.line
  File "/usr/local/lib/python3.9/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/local/lib/python3.9/linecache.py", line 30, in getline
    lines = getlines(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 46, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/local/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
RecursionError: maximum recursion depth exceeded while calling a Python object

I really need help, I have to restart my server every 2 days and it's getting annoying, I can't put a try catch or something like that to prevent everything from crashing?

[miguelgrinberg]

If this sections of the log that you shared the beginning of the problem, or is it a section in the middle? I need to see how it all started, right from the first non-standard message or error.

[MathiasDrapier]

@miguelgrinberg Thank you very much for taking the time to help me, here is more detail on the logs I was able to collect:

I have 10,000 rows repeating with:
2023-03-04 03:17:50,860 - root - WARNING - Cannot send to sid DD9Zi8i8DJKIDNH0AUOC
with different sid.

After I get :

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 790, in _handle_event
    environ = self.server.get_environ(sid, namespace=namespace)
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1519, in findCaller
    f = currentframe()
  File "/usr/local/lib/python3.9/logging/__init__.py", line 160, in <lambda>
    currentframe = lambda: sys._getframe(3)
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1187, in emit
    StreamHandler.emit(self, record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 671, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 621, in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
  File "/usr/local/lib/python3.9/traceback.py", line 103, in print_exception
    for line in TracebackException(
  File "/usr/local/lib/python3.9/traceback.py", line 517, in __init__
    self.stack = StackSummary.extract(
  File "/usr/local/lib/python3.9/traceback.py", line 366, in extract
    f.line
  File "/usr/local/lib/python3.9/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/local/lib/python3.9/linecache.py", line 30, in getline
    lines = getlines(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 46, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/local/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
RecursionError: maximum recursion depth exceeded while calling a Python object
2023-03-04 03:17:51,005 - root - WARNING - Cannot send to sid 4wx0Ns-ZY25pACs8AUN1
2023-03-04 03:17:51,007 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

During handling of the above exception, another exception occurred:

RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 232, in send
    socket.send(packet.Packet(packet.MESSAGE, data=data))
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 85, in send
    if not self.check_ping_timeout():
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 79, in check_ping_timeout
    self.close(wait=False, abort=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 132, in close
    self.server._trigger_event('disconnect', self.sid, run_async=False)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 624, in _trigger_event
    self.logger.exception(event + ' handler error')
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1187, in emit
    StreamHandler.emit(self, record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 671, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 621, in formatException
    traceback.print_exception(ei[0], ei[1], tb, None, sio)
  File "/usr/local/lib/python3.9/traceback.py", line 103, in print_exception
    for line in TracebackException(
  File "/usr/local/lib/python3.9/traceback.py", line 517, in __init__
    self.stack = StackSummary.extract(
  File "/usr/local/lib/python3.9/traceback.py", line 366, in extract
    f.line
  File "/usr/local/lib/python3.9/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/local/lib/python3.9/linecache.py", line 30, in getline
    lines = getlines(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 46, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/lib/python3.9/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
  File "/usr/local/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
RecursionError: maximum recursion depth exceeded while calling a Python object
2023-03-04 03:17:51,009 - root - WARNING - Cannot send to sid 4wx0Ns-ZY25pACs8AUN1
2023-03-04 03:17:51,009 - root - WARNING - Cannot send to sid REslDWH73-jyFURwAUN4
2023-03-04 03:17:51,011 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
2023-03-04 03:17:51,013 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
2023-03-04 03:17:51,015 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
2023-03-04 03:17:51,017 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
2023-03-04 03:17:51,018 - root - ERROR - disconnect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 633, in _get_socket
    s = self.sockets[sid]
KeyError: '4wx0Ns-ZY25pACs8AUN1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 227, in send
    socket = self._get_socket(sid)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 635, in _get_socket
    raise KeyError('Session not found')
KeyError: 'Session not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 807, in _handle_eio_disconnect
    self._handle_disconnect(eio_sid, n)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 709, in _handle_disconnect
    self._trigger_event('disconnect', namespace, sid)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 756, in _trigger_event
    return self.handlers[namespace][event](*args)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 282, in _handler
    return self._handle_event(handler, message, namespace, sid,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 828, in _handle_event
    ret = handler(*args)
  File "/usr/src/app/src/socketManager.py", line 104, in handle_disconnect
    emit('message', str(hcd), to=room)
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 901, in emit
    return socketio.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/flask_socketio/__init__.py", line 462, in emit
    self.server.emit(event, *args, namespace=namespace, to=to,
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 319, in emit
    self.manager.emit(event, data, namespace, room=room,
  File "/usr/local/lib/python3.9/site-packages/socketio/base_manager.py", line 172, in emit
    self.server._emit_internal(eio_sid, event, data, namespace, id)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 644, in _emit_internal
    self._send_packet(eio_sid, self.packet_class(
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 654, in _send_packet
    self.eio.send(eio_sid, encoded_packet)
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 230, in send
    self.logger.warning('Cannot send to sid %s', sid)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1587, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1556, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
  File "/usr/local/lib/python3.9/logging/__init__.py", line 308, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.abc.Mapping)
  File "/usr/local/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

Lots of posts like this...

Then all this repeats ... New sequence of "Cannot send to sid ...." then the stack trace....

[MathiasDrapier]

Very good thank you very much, I will fix this and I will be able to say if the problem persists within 2-3 days

[MathiasDrapier]

Hi @miguelgrinberg, I implemented the fix, but when I restart my server I have a lot of logs like this:

2023-03-07 09:41:04,173 - root - ERROR - post request handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 431, in handle_request
    socket.handle_post_request(environ)
  File "/usr/local/lib/python3.9/site-packages/engineio/socket.py", line 124, in handle_post_request
    p = payload.Payload(encoded_payload=body)
  File "/usr/local/lib/python3.9/site-packages/engineio/payload.py", line 13, in __init__
    self.decode(encoded_payload)
  File "/usr/local/lib/python3.9/site-packages/engineio/payload.py", line 44, in decode
    raise ValueError('Too many packets in payload')
ValueError: Too many packets in payload
2023-03-07 09:41:54,260 - root - ERROR - message handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/engineio/server.py", line 622, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 789, in _handle_eio_message
    self._handle_connect(eio_sid, pkt.namespace, pkt.data)
  File "/usr/local/lib/python3.9/site-packages/socketio/server.py", line 680, in _handle_connect
    'connect', namespace, sid, self.environ[eio_sid])
KeyError: '7dhnhnw0hzbV5svsAAT_'

What are the reasons for this error? I also have a lot of lines like this:
2023-03-07 09:40:39,445 - root - WARNING - None is not connected to namespace /

But all my clients are working fine! I don't understand

[miguelgrinberg]

Did these happen immediately after the restart? It could be that your clients did not realize that the server was restarted and continue sending requests on the old connections. If the error stopped after a few seconds then that is all this is.

[MathiasDrapier]

Yes I thought of that too but the error continues ... My container has been running for 40 minutes and my log file is flooded by these errors. Is it possible that someone ddos me? Maybe that's why these errors happen randomly?
I've been restarting my server for 4 hours and getting these errors over and over.
My log file is already huge, I have thousands of rows per second

[miguelgrinberg]

I'd say you need to identify which client is sending these requests that fail. If it is your own client, then you can then debug that client and hopefully figure it out. If it isn't your own client, then block the IP address and move on.