Keep all I/O on main thread by uatuko · Pull Request #24 · uatuko/grpcxx

uatuko · 2024-01-31T21:37:43Z

Run only one libuv event loop for I/O and move processing to worker threads. This brings a few benefits.

Increased throughput (pushing beyond 400k req/s)
Gives flexibility to run either in a single thread or use multiple threads
Better portability since we no longer need the *nix specific socket duplication

Benchmarks

	1a	1b	2a	2b	3a	3b
single-threaded (`86e43df`)	40k	146k	98k	243k	83k	184k
multi-threaded, hardware concurrency (`e48393d`)	32k	134k	105k	387k	84k	242k
multi-threaded, 2 workers (`e48393d`)	33k	138k	107k	364k	85k	260k
multi-threaded, hardware concurrency (`832d6f3`)	32k	132k	107k	408k	88k	313k
multi-threaded, 2 workers (`832d6f3`)	32k	136k	108k	400k	81k	335k

Benchmark details

Benchmarks were run on a MacBook Pro 2021 (M1 Max, 32GB), macOS 14.2.1 (23C71).

single-threaded (86e43df)

single-threaded (`86e43df`)

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 2.52s, 39615.16 req/s, 2.30MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       16us       276us        23us         4us    86.03%
time for connect:     6.11ms      6.11ms      6.11ms         0us   100.00%
time to 1st byte:     6.28ms      6.28ms      6.28ms         0us   100.00%
req/s           :   39615.21    39615.21    39615.21        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 683.37ms, 146333.61 req/s, 8.51MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       47us       202us        61us         9us    90.71%
time for connect:     6.70ms      6.70ms      6.70ms         0us   100.00%
time to 1st byte:     6.91ms      6.91ms      6.91ms         0us   100.00%
req/s           :  146334.38   146334.38   146334.38        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.02s, 97813.48 req/s, 5.69MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       16us     19.30ms        87us        68us    95.77%
time for connect:    21.52ms     40.72ms     34.48ms      5.81ms    60.00%
time to 1st byte:    31.68ms     40.83ms     36.60ms      3.82ms    40.00%
req/s           :    9782.45     9846.59     9808.98       19.64    70.00%

2b

❯ h2load --clients=10 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 411.38ms, 243087.21 req/s, 14.14MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       52us      9.53ms       331us       158us    77.62%
time for connect:    31.76ms     41.05ms     36.40ms      3.33ms    60.00%
time to 1st byte:    35.13ms     41.30ms     37.84ms      2.97ms    60.00%
req/s           :   24308.96    28455.80    26512.67     1747.30    50.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.21s, 82918.60 req/s, 4.83MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       19us     27.25ms       883us       741us    97.23%
time for connect:    29.08ms    147.25ms     88.74ms     37.85ms    59.00%
time to 1st byte:    32.27ms    173.46ms    109.80ms     45.66ms    59.00%
req/s           :     829.43      873.61      840.39        9.39    75.00%

3b

❯ h2load --clients=100 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello 
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 542.41ms, 184363.74 req/s, 10.73MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       57us     86.61ms      3.73ms      2.99ms    98.76%
time for connect:    88.34ms    165.24ms    131.90ms     22.19ms    58.00%
time to 1st byte:    90.72ms    180.87ms    152.85ms     29.01ms    77.00%
req/s           :    1847.64     1876.05     1865.42        6.07    61.00%

multi-threaded, hardware concurrency (e48393d)

multi-threaded, hardware concurrency (`e48393d`)

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.08s, 32495.28 req/s, 1.89MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       20us       215us        28us         4us    88.19%
time for connect:     6.41ms      6.41ms      6.41ms         0us   100.00%
time to 1st byte:     6.56ms      6.56ms      6.56ms         0us   100.00%
req/s           :   32495.32    32495.32    32495.32        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 749.06ms, 133501.19 req/s, 7.77MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       54us       227us        68us         9us    91.74%
time for connect:     6.66ms      6.66ms      6.66ms         0us   100.00%
time to 1st byte:     6.89ms      6.89ms      6.89ms         0us   100.00%
req/s           :  133501.69   133501.69   133501.69        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 952.59ms, 104977.40 req/s, 6.11MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       21us      9.87ms        74us        37us    93.41%
time for connect:    31.16ms     40.86ms     36.12ms      3.43ms    60.00%
time to 1st byte:    34.75ms     41.04ms     37.27ms      3.21ms    60.00%
req/s           :   10499.21    10511.87    10506.00        3.80    70.00%

2b

❯ h2load --clients=10 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 258.33ms, 387094.28 req/s, 22.52MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       55us      1.85ms       193us        64us    86.81%
time for connect:     4.57ms     20.06ms     14.34ms      5.32ms    50.00%
time to 1st byte:     4.86ms     21.89ms     16.07ms      5.63ms    50.00%
req/s           :   38723.97    38895.01    38798.79       44.69    80.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.19s, 83931.20 req/s, 4.89MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       54us     74.48ms       784us      1.11ms    99.92%
time for connect:    76.11ms    143.72ms    118.79ms     18.98ms    63.00%
time to 1st byte:    78.43ms    177.87ms    142.01ms     31.82ms    55.00%
req/s           :     839.56      841.34      840.51        0.39    70.00%

3b

❯ h2load --clients=100 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 413.92ms, 241592.58 req/s, 14.07MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:      195us      3.71ms      1.76ms       461us    67.38%
time for connect:     6.26ms    161.86ms     79.25ms     46.65ms    60.00%
time to 1st byte:     9.87ms    165.79ms     83.90ms     46.22ms    60.00%
req/s           :    2415.94     2432.61     2424.17        5.01    58.00%

multi-threaded, 2 workers (e48393d)

multi-threaded, 2 workers (`e48393d`)

diff --git a/lib/grpcxx/server.cpp b/lib/grpcxx/server.cpp
index 1d2afcd..39d7a99 100644
--- a/lib/grpcxx/server.cpp
+++ b/lib/grpcxx/server.cpp
@@ -164,7 +164,7 @@ private:
 };
 
 std::hash<std::thread::id> hasher;
-threadpool                 pool(std::thread::hardware_concurrency());
+threadpool                 pool(2);
 
 namespace grpcxx {
 server::server(std::size_t n) noexcept : _handle(), _loop(), _pool(n), _services() {

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.07s, 32541.49 req/s, 1.89MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       20us       246us        28us         5us    84.95%
time for connect:     6.74ms      6.74ms      6.74ms         0us   100.00%
time to 1st byte:     6.89ms      6.89ms      6.89ms         0us   100.00%
req/s           :   32541.52    32541.52    32541.52        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 725.03ms, 137924.57 req/s, 8.02MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       54us       260us        66us         7us    90.25%
time for connect:     2.91ms      2.91ms      2.91ms         0us   100.00%
time to 1st byte:     3.07ms      3.07ms      3.07ms         0us   100.00%
req/s           :  137925.30   137925.30   137925.30        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 920.13ms, 108680.41 req/s, 6.32MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       19us     11.60ms        71us        41us    96.72%
time for connect:    26.25ms     37.74ms     33.33ms      4.05ms    50.00%
time to 1st byte:    31.43ms     37.91ms     34.65ms      3.37ms   100.00%
req/s           :   10868.08    10880.68    10875.67        3.83    70.00%

2b

❯ h2load --clients=10 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 274.91ms, 363750.12 req/s, 21.16MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       53us      6.27ms       196us        74us    93.93%
time for connect:    28.12ms     38.44ms     34.65ms      3.86ms    90.00%
time to 1st byte:    28.35ms     38.77ms     35.68ms      4.02ms    90.00%
req/s           :   36393.14    36486.20    36434.89       30.14    60.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.18s, 84679.10 req/s, 4.93MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       54us     70.57ms       760us      1.09ms    99.88%
time for connect:    72.51ms    150.63ms    120.79ms     20.09ms    66.00%
time to 1st byte:    74.84ms    178.99ms    143.79ms     32.05ms    62.00%
req/s           :     847.16      849.24      848.26        0.49    67.00%

3b

❯ h2load --clients=100 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 385.69ms, 259276.93 req/s, 15.09MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       85us      3.84ms      1.61ms       392us    69.11%
time for connect:     6.14ms    156.10ms     76.65ms     44.63ms    60.00%
time to 1st byte:     9.05ms    160.17ms     81.35ms     44.30ms    61.00%
req/s           :    2592.79     2610.40     2602.33        5.28    56.00%

multi-threaded, hardware concurrency (832d6f3)

multi-threaded, hardware concurrency (`832d6f3`)

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.14s, 31853.81 req/s, 1.85MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       21us       165us        29us         4us    88.96%
time for connect:     6.63ms      6.63ms      6.63ms         0us   100.00%
time to 1st byte:     6.81ms      6.81ms      6.81ms         0us   100.00%
req/s           :   31853.85    31853.85    31853.85        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 759.83ms, 131608.21 req/s, 7.66MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       54us       383us        69us        10us    93.36%
time for connect:     7.22ms      7.22ms      7.22ms         0us   100.00%
time to 1st byte:     7.49ms      7.49ms      7.49ms         0us   100.00%
req/s           :  131608.76   131608.76   131608.76        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 931.64ms, 107337.94 req/s, 6.25MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       21us     17.58ms        73us        58us    98.94%
time for connect:    20.50ms     37.93ms     32.11ms      5.29ms    60.00%
time to 1st byte:    28.22ms     38.10ms     34.00ms      3.67ms    50.00%
req/s           :   10735.09    10745.78    10740.79        3.44    60.00%

2b

❯ h2load --clients=10 --requests=250000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 250000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 612.37ms, 408253.25 req/s, 23.75MB/s
requests: 250000 total, 250000 started, 250000 done, 250000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 250000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 14.55MB (15252160) total, 488.40KB (500120) headers (space savings 94.74%), 7.39MB (7750000) data
                     min         max         mean         sd        +/- sd
time for request:       59us     10.32ms       194us       145us    99.80%
time for connect:     8.88ms     14.66ms     11.20ms      2.34ms    80.00%
time to 1st byte:    12.01ms     20.75ms     16.94ms      3.70ms    40.00%
req/s           :   40837.17    40886.41    40857.02       15.82    60.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.13s, 88172.92 req/s, 5.13MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       55us     22.57ms       713us       659us    96.72%
time for connect:    23.76ms    131.29ms     85.51ms     36.18ms    51.00%
time to 1st byte:    27.03ms    153.26ms    103.56ms     41.91ms    50.00%
req/s           :     882.11      883.91      882.91        0.38    61.00%

3b

❯ h2load --clients=100 --requests=250000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 250000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 799.37ms, 312744.72 req/s, 18.20MB/s
requests: 250000 total, 250000 started, 250000 done, 250000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 250000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 14.55MB (15256000) total, 489.45KB (501200) headers (space savings 94.72%), 7.39MB (7750000) data
                     min         max         mean         sd        +/- sd
time for request:      131us     47.15ms      1.82ms      2.06ms    99.66%
time for connect:    46.26ms    123.88ms     93.67ms     18.31ms    69.00%
time to 1st byte:    51.07ms    179.70ms    127.83ms     41.30ms    58.00%
req/s           :    3128.24     3139.42     3134.10        3.17    57.00%

multi-threaded, 2 workers (832d6f3)

diff --git a/examples/helloworld/main.cpp b/examples/helloworld/main.cpp
index debc569..a04f98d 100644
--- a/examples/helloworld/main.cpp
+++ b/examples/helloworld/main.cpp
@@ -35,7 +35,7 @@ int main() {
 	GreeterImpl greeter;
 	Service     service(greeter);

-	grpcxx::server server;
+	grpcxx::server server(2);
 	server.add(service);

 	std::printf("Listening on [127.0.0.1:7000] ...\n");

multi-threaded, multi-threaded, 2 workers (`832d6f3`)

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.16s, 31633.86 req/s, 1.84MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       21us       151us        29us         4us    92.13%
time for connect:     6.54ms      6.54ms      6.54ms         0us   100.00%
time to 1st byte:     6.70ms      6.70ms      6.70ms         0us   100.00%
req/s           :   31633.89    31633.89    31633.89        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 737.21ms, 135645.82 req/s, 7.89MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6100749) total, 195.32KB (200012) headers (space savings 94.74%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       53us       201us        67us        10us    92.74%
time for connect:     6.23ms      6.23ms      6.23ms         0us   100.00%
time to 1st byte:     6.42ms      6.42ms      6.42ms         0us   100.00%
req/s           :  135646.41   135646.41   135646.41        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 925.88ms, 108005.71 req/s, 6.28MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6101120) total, 195.43KB (200120) headers (space savings 94.73%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       20us     21.09ms        72us       147us    99.99%
time for connect:    19.79ms     40.73ms     34.59ms      6.18ms    90.00%
time to 1st byte:    31.20ms     40.89ms     36.86ms      3.61ms    50.00%
req/s           :   10801.69    10817.39    10807.10        5.41    80.00%

2b

❯ h2load --clients=10 --requests=250000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 10 total client(s). 250000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 624.97ms, 400019.20 req/s, 23.27MB/s
requests: 250000 total, 250000 started, 250000 done, 250000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 250000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 14.55MB (15252160) total, 488.40KB (500120) headers (space savings 94.74%), 7.39MB (7750000) data
                     min         max         mean         sd        +/- sd
time for request:       61us      4.96ms       194us        48us    80.73%
time for connect:    35.18ms     41.63ms     37.97ms      3.16ms    60.00%
time to 1st byte:    35.57ms     41.94ms     38.76ms      3.31ms   100.00%
req/s           :   40009.79    40059.83    40032.54       15.36    60.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.24s, 80826.83 req/s, 4.71MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.82MB (6104700) total, 196.48KB (201200) headers (space savings 94.71%), 2.96MB (3100000) data
                     min         max         mean         sd        +/- sd
time for request:       56us     51.20ms       726us      1.11ms    99.92%
time for connect:    52.87ms    129.97ms    103.99ms     22.56ms    58.00%
time to 1st byte:    54.91ms    180.55ms    134.82ms     41.15ms    57.00%
req/s           :     808.51      810.03      809.33        0.38    65.00%

3b

❯ h2load --clients=100 --requests=250000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --data=examples/helloworld/testdata/hello.grpc-lpm.data http://localhost:7000/helloworld.v1.Greeter/Hello
starting benchmark...
spawning thread #0: 100 total client(s). 250000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 746.30ms, 334985.48 req/s, 19.50MB/s
requests: 250000 total, 250000 started, 250000 done, 250000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 250000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 14.55MB (15256000) total, 489.45KB (501200) headers (space savings 94.72%), 7.39MB (7750000) data
                     min         max         mean         sd        +/- sd
time for request:       76us     22.80ms      1.70ms      1.16ms    97.92%
time for connect:    24.26ms    134.07ms     85.76ms     33.46ms    49.00%
time to 1st byte:    26.26ms    175.94ms    113.76ms     48.58ms    49.00%
req/s           :    3350.28     3363.97     3357.50        3.88    58.00%

A scheduler to schedule and run tasks either using multiple threads or in a single thread.

uatuko added 4 commits January 31, 2024 21:34

(grpcxx) quick hack to keep all io on main thread

5052c5e

WIP: single-threaded baseline for benchmarking

86e43df

(grpcxx) still a hack but more work offloaded to worker threads

e48393d

(grpcxx) tidy-up

dc14222

uatuko mentioned this pull request Feb 1, 2024

Asio #23

Closed

uatuko added 4 commits February 1, 2024 14:55

(grpcxx) drop worker pool implementation

9d3f587

(grpcxx) scheduler

a92c9e3

A scheduler to schedule and run tasks either using multiple threads or in a single thread.

(grpcxx) tidy-up

693e2b2

(grpcxx) tidy-up

832d6f3

uatuko marked this pull request as ready for review February 1, 2024 17:02

uatuko added 3 commits February 1, 2024 17:06

(grpcxx) tidy-up

e323a29

(docs) update api docs

8d80711

(docs) tidy-up

82b6e30

uatuko merged commit 30e3811 into main Feb 2, 2024

uatuko deleted the feature/main-thread-io branch February 2, 2024 11:50

uatuko mentioned this pull request Feb 4, 2024

Reduce dynamic memory allocation when writing TCP data #6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep all I/O on main thread#24

Keep all I/O on main thread#24
uatuko merged 11 commits intomainfrom
feature/main-thread-io

uatuko commented Jan 31, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

uatuko commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmark details

single-threaded (86e43df)

1a

1b

2a

2b

3a

3b

multi-threaded, hardware concurrency (e48393d)

1a

1b

2a

2b

3a

3b

multi-threaded, 2 workers (e48393d)

1a

1b

2a

2b

3a

3b

multi-threaded, hardware concurrency (832d6f3)

1a

1b

2a

2b

3a

3b

multi-threaded, multi-threaded, 2 workers (832d6f3)

1a

1b

2a

2b

3a

3b

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

uatuko commented Jan 31, 2024 •

edited

Loading

single-threaded (`86e43df`)

multi-threaded, hardware concurrency (`e48393d`)

multi-threaded, 2 workers (`e48393d`)

multi-threaded, hardware concurrency (`832d6f3`)

multi-threaded, multi-threaded, 2 workers (`832d6f3`)