dealing w/ HTTP non-compliant PVE API when proxying through nginx

[bdruth]

Just a PSA / heads-up / request for roast, if wrong ... I recently setup my provisioning to use the PVE API through my nginx proxy and made a disconcerting and, frankly, enraging discovery. The provider makes certain requests to PVE that (knowingly?) result in a 500 response, in my case, querying /api2/json/cluster/ha/resources/103 when I don't have HA, for an LXC. This is not the source of my rage - I assume this is like this for a reason, it is what it is. However, PVE responds with

HTTP/1.1 500 no such resource 'ct:103'
Cache-Control: max-age=0
Connection: close
Date: Wed, 27 Nov 2024 18:42:16 GMT
Pragma: no-cache
Server: pve-api-daemon/3.0
Content-Length: 13
Content-Type: application/json;charset=UTF-8
Expires: Wed, 27 Nov 2024 18:42:16 GMT

{"data":null}

Now, I'm not an RFC expert, but it's my understanding that the start-line having the extra no such resource 'ct:103' is a no-no ... maybe in particular because of the 'ct:103'? In any case, nginx marks the upstream as temporarily unavailable when it sees this response (upstream server temporarily disabled while reading response header from upstream), and subsequent requests fail with a 502 gateway error because there aren't any upstreams available to handle the request (no live upstreams while connecting to upstream).

Net effect? This renders the provider unusable. The workaround? Well, this took me a frustratingly long time to figure out, because I'm not an nginx guru, but you can define the upstream { ... } block in your nginx configuration, and on the server ... line, you can append max_fails=0 which effectively tells nginx to never disable the backend. This ends up looking like this:

upstream proxmox {
  server 10.0.0.2:8006 max_fails=0;
}

You then use this upstream in your proxy_pass configuration line, e.g.: proxy_pass https://proxmox/;

As I was trying to sort this, I posted to the proxmox forum ... hoping someone else that had used nginx would have seen this / solved this before, but as of this writing, no responses ... I haven't found anyplace to actually post bugs or issues for the PVE API, their github projects are read-only replicas of their git infrastructure, and so far I've only found pointers to the forum. If anyone knows where it would be appropriate to post the details of this issue, I'm happy to do so.

[Tinyblargon]

@bdruth Proxmox uses this site for their bug tracking https://bugzilla.proxmox.com/index.cgi
You do need an account to use it. Keep in mind that the email address of the account is publicly displayed, so maybe make a dedicated one for it.

[bdruth]

Thank you! I'll get something posted over there if I can't find an existing issue.

[bdruth]

Created https://bugzilla.proxmox.com/show_bug.cgi?id=5951

[Fabian-Gruenbichler]

FWIW - I don't think our API is non-compliant here (although it might be a bit unusual). I don't know why nginx chokes on the custom status line/reason phrase - those have always been a thing, and any sane client either displays them or ignores them, the status code is the only thing that has a semantic meaning that can be matched upon. See my response on our bug tracker for more details and a possible workaround.

I now think the root cause is just that we return 500 here (while 404 would be more appropriate, see https://bugzilla.proxmox.com/show_bug.cgi?id=5698), which nginx in its default configuration considers a sign of the upstream failing, but that is configurable (at the risk of not handling actually fatal 500s then):

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream

[bdruth]

@Fabian-Gruenbichler I can't get proxy_next_upstream to ignore the error that's happening, tried setting it to just timeout as well as just http_404 ... in both cases, the error log reflects the same message as reported above, which leads me to believe that this is a parsing error of some sort, not because of the 500. Additional support for this comes from the default indicated in the link you provided:

Default:	proxy_next_upstream error timeout;

if this is truly the default, then, per the documentation, a 5xx error should not be considered an unsuccessful attempt.

Note

The directive also defines what is considered an unsuccessful attempt of communication with a server. The cases of error, timeout and invalid_header are always considered unsuccessful attempts, even if they are not specified in the directive. The cases of http_500, http_502, http_503, http_504, and http_429 are considered unsuccessful attempts only if they are specified in the directive.

[Tinyblargon]

In some situations, the upstream SDK we use attempts to get items that might not exist. The reason for this is that returning every resource costs a lot of data for bigger instances.

[bdruth]

totally makes sense, thx!