Requests not reaching server when using the DO Toronto Region with a url and HTTPS

[coreyja]

Hey Team!

I've got a bit of a weird one to report today! This afternoon I realized that my snake Hovering Hobbs [hosted in Digital Ocean Toronto] had been losing games for a couple days straight and appeared to be down. I kicked my server and realized the problem was persisting

Pulled up my Honeycomb reporting and saw something interesting, no requests were making it to Hobbs since 6/14 ~00:00 UTC[Just converted that to UTC to post and that 00:00 UTC is suspicious for sure lol]

I dug in and here is what I think I know so far!

Requests are not reaching Hobbs's server [1] [hosted in DO Toronto], IFF the battlesnake region is set to DO Toronto. If I set to any other region requests work fine. Furthermore this only happens over HTTPS with a fully qualified url. If I use the http://ip OR http://url requests come through. But doing https://url is broken

[1] As far as I can tell, I don't have super detailed logging at the lowest level in my Caddy reverse proxy server, but I'm not seeing requests make it to my Rust Server

[coreyja]

This is NOT urgent for me, I just turned off my http -> https redirects and pointed Hobbs to the http url

But I don't know if this could be affecting other snakes as well

[bvanvugt]

I wonder if the redirects are not being followed correctly in that region?

[bvanvugt]

(or i might be misreading the details here)

[coreyja]

I think I caused the confusion by mentioning the redirects I turned off. That was really just an implementation detail of my server. I had http -> https redirects turned before so I needed to make changes to allow http traffic through without redirecting to https

But when I set my snakes to https on the snake detail page Im seeing this behavior, and shouldn’t have any redirects involved

[bvanvugt]

Got it 👍. I'll tee this up for @robbles to dig in next week :)

[coreyja]

Just adding more information!

Bultz in discord can confirm the same issue with https and the Toronto region, HOWEVER their snake is hosted in their homelab. So being hosted in the Toronto region might not be related

[robbles]

Hey @coreyja! I looked into our logs and prodded the Digital Ocean app that's handling these requests, and was able to reproduce the issue.

It looks to me like some kind of odd infrastructure issue, since the other DO regions are not affected, but are running the exact same Docker image. I redeployed the app, and the problem seems to have resolved itself. Can you confirm whether one of your snakes works now with a HTTPS URL?

[coreyja]

Yup I can confirm that using an HTTPS url is working again for me in the Toronto region!

Thanks for digging in @robbles!