add HTTP version selection API

[StefanKarpinski]

In order to properly handle JuliaLang/Downloads.jl#84 we need a NetworkOptions option for mapping hosts (full URLs with host extracted probably) to HTTP versions (1, 2, 3). This sounds simple enough and the first crack I took at this was to have these environment variables:

• JULIA_HTTP1_HOSTS
• JULIA_HTTP2_HOSTS
• JULIA_HTTP3_HOSTS

Each of these takes a host pattern value like the JULIA_*_VERIFY_HOSTS variables. Then the question is if a host matches multiple of these variables, do you pick the lowest or highest HTTP version? My initial thought was to take the lowest. But then this kind of situation occurred to me:

• you want to use HTTP/1.1 for **.github.com
• you want to use HTTP/3 for **.google.com
• you want to use HTTP/2 for everything else

Intuitively, one wants to write this something like this:

export JULIA_HTTP1_HOSTS="**.github.com"
export JULIA_HTTP2_HOSTS="**"
export JULIA_HTTP3_HOSTS="**.google.com"

Of course, with the "least matching" rule, that doesn't work as intended: the **.google.com pattern for HTTP/3 has no effect since every host matches the ** pattern for HTTP/2. We could say that we match in this order: HTTP/1, then HTTP/3, then HTTP/2, but that feels very as hoc. What if HTTP/4 gets developed? Where does that go in that pattern?

[DilumAluthge]

You could let the user customize the match order. E.g. maybe by default we match in the order 1, 3, 2. But the user can provide:

export JULIA_HTTP_VERSIONS_PREFER="2,3,1"

And then we match in that order.

Any HTTP versions that aren't in the list that the user provides in JULIA_HTTP_VERSIONS_PREFER (e.g. in the future when HTTP/4 is released) get appended to the end of the list. So if the user writes this:

export JULIA_HTTP_VERSIONS_PREFER="2,3"

Since they didn't put HTTP/1.1 in the match order list, we'll tack it on to the end, i.e. we'll behave as if:

JULIA_HTTP_VERSIONS_PREFER="2,3,1"

And proceed to match in that order.

[StefanKarpinski]

That approach doesn't allow preferring different HTTP versions for specific hosts which seems important for this. The original motivation is gone because (IIRC), the issue was that libcurl has a bug in it's HTTP/2 implementation where the User-Agent header was not getting set on each request; meanwhile GitHub refuses downloads that don't have the User-Agent header set. But in that kind of a situation, we would want to disable HTTP/2 only for GitHub, no everywhere: on any site that doesn't demand a User-Agent header (why?), everything would work fine. Similarly, it would make sense to only enable HTTP/3 on Google properties since they're the only ones previewing that functionality and most domains are not going to respond if we send them UDP packets.

[StefanKarpinski]

I no longer think this is necessary. The reason we needed to force HTTP/1 for github.com turned out to be because there was a bug in libcurl where using the user agent option failed to set the user agent header on all requests when using HTTP/2, see curl/curl#6312 for the bug and JuliaLang/Downloads.jl#88 for the workaround (setting the user agent as a header as well). Since github.com downloads require having the user agent set, some requests were failing when using HTTP/2. This bug has since been fixed in libcurl and worked around in Downloads, so we no longer need to select specific HTTP versions for specific domains.