Benchmarking etc...

[tomchristie]

Putting this up here as a reminder that:

• At some point we ought to have a case study or something, demo'ing when/why async is useful.
• We might want to do benchmarking to evaluate any possible areas for improvement.

Eg. https://blog.jonlu.ca/posts/async-python-http

Most likely our perf. vs aiohttp appears poor because they're using DNS caching, which we're not. But that hypothesis would need digging into further.

[tomchristie]

Also: https://www.scrapingbee.com/blog/best-python-http-clients/

[dalf]

I've updated to pyhttp-benchmark (benchmark of http clients with a Caddy server with TLS, it is based on https://github.com/florimondmanca/httpxprof ).

It seems anyio is slower then the asyncio backend when there are a lot of data to download, but not when there are a lot of small requests.

See:

• scenarios.py
• cases ( httpcore._async.http2 and stringprep are manually imported to avoid import during the benchmark )

[tomchristie]

Ah some great stuff in there, thanks.

I found it a bit difficult trying to parse through all the data you've collated. Some thoughts on that...

• In this context I wouldn't want to be interested in HTTP/2 at all. We really only want to compare like-for-like. There's a place for benchmarking HTTP/1.1 vs HTTP/2 in different contexts, but it really is an entirely separate discussion. None of the other libraries support HTTP/2, and we don't enable it by default, so really it just adds noise to the data.
• I probably wouldn't really be interested in curio in this context.
• I probably wouldn't really be interested in httpcore in this context. If we're not handling content-decoding, then it's not a like-for-like comparison, and not really useful/valid.
• I wouldn't really be interested in uvloop comparisons. It adds extra noise, and we'll have more meaningful numbers if what we're looking at is like-for-like. Eg. aiohttp vs. httpx-on-asyncio (Without uvloop for either).

Really the important ones to me would be:

• aiohttp
• requests
• httpx
• httpx running on asyncio
• httpx running on trio

[dalf]

See the like-for-like branch: https://github.com/dalf/pyhttp-benchmark/blob/like-for-like/results/output.md

Before drawing conclusions, I think the code of the cases need a review: https://github.com/dalf/pyhttp-benchmark/tree/like-for-like/src/pyhttpbenchmark/cases

Also about "httpx running on trio" do mean

• trio.run(main_args_kwargs) (current implementation)
• or anyio.run(main_args_kwargs, backend='trio')
  ?

[tomchristie]

I mean trio.run(main_args_kwargs). I strongly assume there'd be absolutely no measurable difference between the two cases.

I'm not sure I quite understand the point of anyio.run(). I mean, actually I do get that it makes anyio be a more complete set of primitives in itself, but in a different way it also it slightly confuses the issue.

[tomchristie]

See the like-for-like branch

Oh, that's way neater. Nice work!

The one that's missing is standard sync HTTPX. (Which is interesting as a point of comparison vs. requests)

[tomchristie]

Also I don't think the "anyio" case is a useful comparison here. I'd assumed that httpx-asyncio vs. http-anyio meant using a different backend for the client in each case, but that's not what it means here.

Actually in these cases it's to do with "if you write the client code in anyio" vs. "if you write the client code in asyncio". It'd really only make sense to measure "anyio" here if we were comparing httpx-anyio vs. aiohttp-anyio.