Poor service recovery after restarts/deploys

[dyc3]

Current Behavior

Sometimes, deploys result in clients not reconnecting correctly. Eventually the system recovers, assisted by manual page refreshes, and playback is not interrupted.

Expected Behavior

Service restarts and deploys should result in clients reconnecting to the room gracefully. Video playback should remain uninterrupted.

Steps To Reproduce

Precise steps unknown

1. Watch a video on the staging branch
2. Have more than 1 user in the room
3. While the video is playing, perform a deploy
4. Note whether or not all clients reconnect and end up in the same instance of the room.

Environment

• This happens on the official site, opentogethertube.com
• This happens using a self-hosted version.
• I'm using the docker image.

Room name or URL

No response

Video URL

No response

Anything else?

No response

[dyc3]

I've found that this only occurs when the monolith is being deployed.

[dyc3]

I'm seeing these logs:

2024-04-09T21:57:42Z app[6e824031ce2087] ewr [info]2024-04-09T21:57:42.270644Z ERROR ott_balancer::balancer: failed to handle client inbound: channel closed
2024-04-09T21:57:42Z app[6e824031ce2087] ewr [info]2024-04-09T21:57:42.270663Z  INFO ott_balancer::balancer: monolith disconnected, stopping client inbound handler
2024-04-09T21:57:43Z app[6e824031ce2087] ewr [info]2024-04-09T21:57:43.944497Z ERROR client_entry{room_name=RoomName("5b333992-dc12-45cd-af60-73d963d94e2e") client_id="5986a053-047f-4192-b8e1-0eda70d45fc0"}: ott_balancer::client: Error sending client message to balancer: channel closed
2024-04-09T21:57:43Z app[6e824031ce2087] ewr [info]2024-04-09T21:57:43.944525Z  INFO client_entry{room_name=RoomName("5b333992-dc12-45cd-af60-73d963d94e2e") client_id="5986a053-047f-4192-b8e1-0eda70d45fc0"}: ott_balancer::client: ending client connection
2024-04-09T21:57:43Z app[6e824031ce2087] ewr [info]2024-04-09T21:57:43.944591Z ERROR ott_balancer::service: Error in websocket connection: channel closed