Milvus Concurrent Connection Issue in Kubernetes

[iamglad22]

Hi, We installed Milvus in Kubernetes cluster with ingress-nginx and Nvidia. We are able to connect in Milvus using our uri but when do concurrent connections, only about 30% of the connections are successful...ex. out of 10 parallel connections, only 3 are successful and 7 fails to get Milvus connection. We followed the processes indicated here:
https://milvus.io/docs/azure.md
https://milvus.io/docs/ingress.md - For ingress

Other Details:
Cloud Provider: Azure
Number of Nodepool: 2 (1 user nodepool with instance-type=Standard_NC4as_T4_v3)
GPU: 1 for index node, 1 for query node
ingress-nginx version=1.11.3
Service Type: ClusterIP
Milvus Image: milvusdb/milvus:v2.5.0-beta
Kubelet Version: v1.30.5
Kube-Proxy Version: v1.30.5
pymilvus version == 2.4.9

Are there any settings that we need to change to allow 100% parallel connection requests to succeed on Milvus on AKS cluster of order 1000, 10000 parallel connection requests?

code snippet to connect to Milvus ...

connections.connect("default",uri="https://:443")
print(utility.list_collections())
['hello_milvus0']

Below is a sample log from proxy-xxxx-xxxx pod:

[2024/12/09 05:42:39.844 +00:00] [INFO] [proxy/impl.go:6169] ["connect received"] [traceID=daed06926f5dbc587e43216] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.827962"] [user=] [host=VM-MILVUS-1] [db=default]
[2024/12/09 05:42:39.846 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=e3dc250535a6f8080bdb] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.827555"] [user=] [host=VM-MILVUS-1] [identifier=4544850715] [last_active_time=2024/12/09 05:42:39.846 +00:00]
[2024/12/09 05:42:39.846 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=daed06926f5dbc587] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.827962"] [user=] [host=VM-MILVUS-1] [identifier=454485071580] [last_active_time=2024/12/09 05:42:39.846 +00:00]
[2024/12/09 05:42:39.846 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=85a4056bdb06a6e] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.827211"] [user=] [host=VM-MILVUS-1] [identifier=454485071580] [last_active_time=2024/12/09 05:42:39.846 +00:00]
[2024/12/09 05:42:39.846 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=6e2f8979b7c7080666] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.827371"] [user=] [host=VM-MILVUS-1] [identifier=45448507158] [last_active_time=2024/12/09 05:42:39.846 +00:00]
[2024/12/09 05:42:39.911 +00:00] [INFO] [proxy/impl.go:6169] ["connect received"] [traceID=bdd0a10105bbea739653e61] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.894120"] [user=] [host=VM-MILVUS-1] [db=default]
[2024/12/09 05:42:39.912 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=bdd0a10105bbea7396] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.894120"] [user=] [host=VM-MILVUS-1] [identifier=45448507159] [last_active_time=2024/12/09 05:42:39.912 +00:00]
[2024/12/09 05:42:39.912 +00:00] [INFO] [proxy/impl.go:6169] ["connect received"] [traceID=b2c92cdbce7379046ed707] [sdk_type=Python] [sdk_version=2.4.3] [local_time="2024-12-09 05:42:39.895823"] [user=] [host=VM-MILVUS-1] [db=default]

Please assist me on how we can fix it.

[haorenfsa]

Are there any settings that we need to change to allow 100% parallel connection requests to succeed on Milvus on AKS cluster of order 1000, 10000 parallel connection requests?

I think no. It supports many parallel connection requests by default.

Here are some possible solutions:

• Do you see any pod crash in your cluster?
• Use pymilvus v2.5 to access Milvus v2.5
• You may also try out Milvus v2.4 stable versions