Make massive requests

[rafaelpuyet]

How can I handle a massive number of requests in less time?

I attempted to check a list of 2k servers using a bottleneck to limit concurrent requests to 50, and it worked. However, at specific times, it gets stuck. I experimented with 100 concurrent requests, but the problem worsened; it got stuck, and connections started to drop. I initially thought it might be a DNS resolver issue, but testing with the same list using direct IP and port yielded the same problem.

I tried adjusting the timeout, but the issue persisted. After inspecting the code, I suspected that the problem might be caused by net.createConnection(), and that at a certain point of open connections, it hangs up. However, I couldn't find any information related to such a problem.

let portal = net.createConnection({ port: port, host: hostname, lookup: customLookup }, async () => {

Any guidance or help would be appreciated. Currently, I can handle 2k requests with 50 concurrent requests, but it still gets stuck for 1-2 seconds at random moments. If I increase the number of requests, it starts throwing timeouts.

[woodendoors7]

That's a lot of requests! Are you running it all on the same thread? I am not sure if node's net library is asynchronous internally, and it might be overloaded with creating so many new connections and waiting for responses.