High latency / Low performance

[ggrandes]

I'm testing ACM-Enclave on a c6g.xlarge machine (it plays the same on a c6i.xlarge).

Both tests (acm/no-acm) have been executed on the same machine, only changing the Apache configuration to use a local files or the pkcs#11 interface.

--- acm/pkcs11 ---
SSLCertificateKeyFile "pkcs11:model=p11ne-token;manufacturer=Amazon;token=httpd-acm-token;id=%01;object=acm-key;type=private?pin-value=xxx"
SSLCertificateFile "/run/nitro_enclaves/acm/httpd-cert-xxx.pem"

--- no-acm/local files ---
SSLCertificateKeyFile "/etc/httpd/enclave.key"
SSLCertificateFile "/etc/httpd/enclave.crt"

Summary:

srv	type	req/s	latency-95%	result
c6g.xlarge	acm	109	34	worst
c6g.xlarge	no-acm	288	12	best
c6i.xlarge	acm	219	15	worst
c6i.xlarge	no-acm	899	2	best

Detail:

Test with: c6g.xlarge + aws-nitro-enclaves-acm-1.2.0-2.amzn2023.aarch64

ACM Enabled

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests

Server Software:        Apache/2.4.56
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   36.434 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      1172000 bytes
HTML transferred:       48000 bytes
Requests per second:    109.79 [#/sec] (mean) <<<<<<<<<<<<<<<<<<<<<<<<<< slow
Time per request:       18.217 [ms] (mean)
Time per request:       9.109 [ms] (mean, across all concurrent requests)
Transfer rate:          31.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       13   18   6.9     14      64
Processing:     0    0   0.3      0       4
Waiting:        0    0   0.2      0       3
Total:         14   18   7.0     14      64

Percentage of the requests served within a certain time (ms)
  50%     14
  66%     16
  75%     22
  80%     22
  90%     28
  95%     34 <<<<<<<<<<<<<<<<<<<<<<<<<< slow
  98%     41
  99%     44
 100%     64 (longest request)

Local files without ACM (self-signed cert, RSA-2048)

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests

Server Software:        Apache/2.4.56
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   13.882 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      1172000 bytes
HTML transferred:       48000 bytes
Requests per second:    288.14 [#/sec] (mean) <<<<<<<<<<<<<<<<<<<<<<<<<< fast
Time per request:       6.941 [ms] (mean)
Time per request:       3.471 [ms] (mean, across all concurrent requests)
Transfer rate:          82.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5    6   2.1      6      22
Processing:     0    1   1.1      0      11
Waiting:        0    0   0.7      0       7
Total:          5    7   2.4      6      22

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      8
  80%      9
  90%     10
  95%     12 <<<<<<<<<<<<<<<<<<<<<<<<<< fast
  98%     14
  99%     16
 100%     22 (longest request)

Test with c6i.xlarge + - aws-nitro-enclaves-acm-1.2.0-1.amzn2023.x86_64

ACM Enabled

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt     
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests

Server Software:        Apache/2.4.56
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   18.214 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      1172000 bytes
HTML transferred:       48000 bytes
Requests per second:    219.61 [#/sec] (mean) <<<<<<<<<<<<<<<<<<<<<<<<<< slow
Time per request:       9.107 [ms] (mean)
Time per request:       4.554 [ms] (mean, across all concurrent requests)
Transfer rate:          62.84 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5    9   2.7      8      33
Processing:     0    0   0.3      0      13
Waiting:        0    0   0.2      0       9
Total:          5    9   2.7      8      34

Percentage of the requests served within a certain time (ms)
  50%      8
  66%      8
  75%      9
  80%      9
  90%     12
  95%     15 <<<<<<<<<<<<<<<<<<<<<<<<<< slow
  98%     19
  99%     21
 100%     34 (longest request)

Local files without ACM (self-signed cert, RSA-2048)

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt         
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests

Server Software:        Apache/2.4.56
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   4.446 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      1172000 bytes
HTML transferred:       48000 bytes
Requests per second:    899.77 [#/sec] (mean) <<<<<<<<<<<<<<<<<<<<<<<<<< fast
Time per request:       2.223 [ms] (mean)
Time per request:       1.111 [ms] (mean, across all concurrent requests)
Transfer rate:          257.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    2   0.1      2       3
Processing:     0    0   0.0      0       1
Waiting:        0    0   0.0      0       1
Total:          2    2   0.1      2       3

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      2
  95%      2 <<<<<<<<<<<<<<<<<<<<<<<<<< fast
  98%      3
  99%      3
 100%      3 (longest request)

[eugkoira]

I did the same test for nginx and I see approximately the same result on c6g.xlarge:

type	req/s	latency-95%	result
c6g.xlarge	acm	159	26
c6g.xlarge	no-acm	258	11

Details

$ cat /etc/nitro_enclaves/allocator.yaml
---
# Enclave configuration file.
#
# How much memory to allocate for enclaves (in MiB).
memory_mib: 512
#
# How many CPUs to reserve for enclaves.
cpu_count: 2

ACM Enabled

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests


Server Software:        nginx/1.22.1
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   25.135 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      972000 bytes
HTML transferred:       48000 bytes
Requests per second:    159.14 [#/sec] (mean)
Time per request:       12.568 [ms] (mean)
Time per request:       6.284 [ms] (mean, across all concurrent requests)
Transfer rate:          37.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   12   5.5     11      65
Processing:     0    0   1.0      0      32
Waiting:        0    0   0.9      0      31
Total:         10   13   5.7     11      65

Percentage of the requests served within a certain time (ms)
  50%     11
  66%     11
  75%     11
  80%     12
  90%     13
  95%     26
  98%     36
  99%     39
 100%     65 (longest request)

Local files without ACM (self-signed cert, RSA-2048)

$ ab -n 4000 -c 2 -s 5 -f TLS1.2 -Z ECDHE-RSA-AES256-GCM-SHA384 https://localhost/test.txt
This is ApacheBench, Version 2.3 <$Revision: 1903618 $>

Benchmarking localhost (be patient)
...
Finished 4000 requests


Server Software:        nginx/1.22.1
Server Hostname:        localhost
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        localhost

Document Path:          /test.txt
Document Length:        12 bytes

Concurrency Level:      2
Time taken for tests:   15.470 seconds
Complete requests:      4000
Failed requests:        0
Total transferred:      972000 bytes
HTML transferred:       48000 bytes
Requests per second:    258.57 [#/sec] (mean)
Time per request:       7.735 [ms] (mean)
Time per request:       3.867 [ms] (mean, across all concurrent requests)
Transfer rate:          61.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5    7   1.9      6      16
Processing:     0    1   1.2      0       6
Waiting:        0    1   1.0      0       6
Total:          5    8   2.0      7      16
WARNING: The median and mean for the waiting time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      7
  66%     10
  75%     10
  80%     10
  90%     10
  95%     11
  98%     11
  99%     11
 100%     16 (longest request)