Skip to content

Add a benchmark for throughput and handshakes using bechamel#500

Merged
hannesm merged 10 commits intomirleft:mainfrom
hannesm:bench
Aug 20, 2024
Merged

Add a benchmark for throughput and handshakes using bechamel#500
hannesm merged 10 commits intomirleft:mainfrom
hannesm:bench

Conversation

@hannesm
Copy link
Copy Markdown
Member

@hannesm hannesm commented Aug 6, 2024

This is based on @reynir's work in MirageVPN

//cc @dinosaure

@hannesm
Add a benchmark for throughput using bechamel
1c98d0d 
This is based on @reynir's work in MirageVPN
@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Aug 6, 2024 •
edited
Loading

We could add:

  • handshake benchmarks (taking different kex into account, FF-DHE, ECDSA, 25519) see fe9fa7c
  • different key types (at the moment RSA is used)
  • different block sizes (at the moment it is 1024 bytes)
  • server side benchmarks (send, receive)

I don't quite understand the output (would be great to have a "throughput in MB/s" and "handshake in number of handshakes per second").

This is a start for conducting some stats with the latest release in order to measure and investigate #497 performance.

Happy to get feedback on the concrete implementation and what matrix would make sense.

@hannesm hannesm changed the title Add a benchmark for throughput using bechamel Add a benchmark for throughput and handshakes using bechamel Aug 6, 2024
@dinosaure
Copy link
Copy Markdown
Contributor

dinosaure commented Aug 19, 2024

I just pushed a commit which adds a new framework to see the throughput of ocaml-tls (à la mirage-crypto). Currently, we test only aes-128-gcm and chacha20-poly1305. These benchmarks should be a mirror of what mirage-crypto can do (means, we should infer a factor between these two libraries).

@hannesm
throughput: conduct more tests (tls1.2, more ciphers)
dda3f5b 
throughput: also add handshake tests

throughput: output markdown tables
@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Aug 19, 2024

@dinosaure I pushed a commit to also conduct the handshake/s in throughput.exe. the same commit extends some more ciphersuites, and finally outputs markdown tables. WDYT?

dinosaure
dinosaure approved these changes Aug 19, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@dinosaure dinosaure left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably can delete the benchmark with bechamel. For the handshake, bechamel can be a little more “accurate” in the sense that it tries to execute the handshake several times and tries to guess the factor that determines the execution time between 10, 20, ..., 100 handshakes. It does this by analyzing the results to remove “noise”, as long as it has no influence on the coefficient of determination.

The proposed throughput framework does not do this, and the results can therefore be influenced by “noise” and influence our interpretation. So, as you wish!

@hannesm
Copy link
Copy Markdown
Member Author

hannesm commented Aug 20, 2024

Thanks for your reply @dinosaure. I'm in favour of having a single module with the benchmarks. We can later always revise.

Indeed, there's noise in the benchmarks. I suspect that we'll at some point get towards less noise :)

@hannesm
separate throughput and handshake benchmarks
167d425 
also add ed25519 and p256 in handshake
@hannesm hannesm merged commit dd3d856 into mirleft:main Aug 20, 2024
@hannesm hannesm deleted the bench branch August 20, 2024 07:08
@hannesm hannesm mentioned this pull request Aug 20, 2024
Merged
hannesm added a commit to hannesm/opam-repository that referenced this pull request Aug 21, 2024
@hannesm
[new release] tls (6 packages) (1.0.0)
894758c 
CHANGES:

* API breaking change: remove usage of Cstruct.t inside of TLS, use bytes
  and string instead (mirleft/ocaml-tls#497 by @art-w, @hannesm, @dinosaure, @reynir)
  Performance is up to 3x improved (bandwidth), 2x improvement for handshake/s
  on an Intel Core(TM) i7-5600U CPU @ 2.60GHz
* FEATURE: add tls-miou-unix package, which adds miou support for TLS
  (mirleft/ocaml-tls#494 mirleft/ocaml-tls#503 @dinosaure)
* FEATURE: tls-lwt and tls-async: allow TLS over an existing connection
  `Tls_lwt.client_of_channels : Tls.Config.client -> ?host:[`host] Domain_name.t -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  and
  `Tls_lwt.server_of_channels : Tls.Config.server -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  (mirleft/ocaml-tls#499 @art-w @MisterDA)
* API breaking changes: revise errors - reduce the polymorphic variant
  in size, align it with RFC specified errors, be in parts more precise
  about errors, in other parts skip data (mirleft/ocaml-tls#505, @hannesm - fixes mirleft/ocaml-tls#491)
  NB: if you relied on a specific error constructor, please open an issue
* Remove unused constructors from Packet.{alert_type, compression_methods,
  client_certificate_type, extension_type} (mirleft/ocaml-tls#505, @hannesm)
  NB: if you relied on specific constructors, please open an issue
* API breaking change: Tls.Config.{server,client} now return a result
  type instead of raising an exception (mirleft/ocaml-tls#502, @hannesm, fixes mirleft/ocaml-tls#411)
* FEATURE: add bench/speed.exe, a benchmark for bandwidth (for different
  ciphersuites) and handshakes (different key exchanges and private keys)
  (mirleft/ocaml-tls#500 @hannesm @dinosaure @reynir)
* BUGFIX: tests/feedback.exe update with TLS 1.3 semantics, run as test
  (mirleft/ocaml-tls#501, @hannesm - reported by @dinosaure)
@hannesm hannesm mentioned this pull request Aug 21, 2024
Merged
hannesm added a commit to hannesm/opam-repository that referenced this pull request Aug 22, 2024
@hannesm
[new release] tls (6 packages) (1.0.0)
cd0112b 
CHANGES:

* API breaking change: remove usage of Cstruct.t inside of TLS, use bytes
  and string instead (mirleft/ocaml-tls#497 by @art-w, @hannesm, @dinosaure, @reynir)
  Performance is up to 3x improved (bandwidth), 2x improvement for handshake/s
  on an Intel Core(TM) i7-5600U CPU @ 2.60GHz
* FEATURE: add tls-miou-unix package, which adds miou support for TLS
  (mirleft/ocaml-tls#494 mirleft/ocaml-tls#503 @dinosaure)
* FEATURE: tls-lwt and tls-async: allow TLS over an existing connection
  `Tls_lwt.client_of_channels : Tls.Config.client -> ?host:[`host] Domain_name.t -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  and
  `Tls_lwt.server_of_channels : Tls.Config.server -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  (mirleft/ocaml-tls#499 @art-w @MisterDA)
* API breaking changes: revise errors - reduce the polymorphic variant
  in size, align it with RFC specified errors, be in parts more precise
  about errors, in other parts skip data (mirleft/ocaml-tls#505, @hannesm - fixes mirleft/ocaml-tls#491)
  NB: if you relied on a specific error constructor, please open an issue
* Remove unused constructors from Packet.{alert_type, compression_methods,
  client_certificate_type, extension_type} (mirleft/ocaml-tls#505, @hannesm)
  NB: if you relied on specific constructors, please open an issue
* API breaking change: Tls.Config.{server,client} now return a result
  type instead of raising an exception (mirleft/ocaml-tls#502, @hannesm, fixes mirleft/ocaml-tls#411)
* FEATURE: add bench/speed.exe, a benchmark for bandwidth (for different
  ciphersuites) and handshakes (different key exchanges and private keys)
  (mirleft/ocaml-tls#500 @hannesm @dinosaure @reynir)
* BUGFIX: tests/feedback.exe update with TLS 1.3 semantics, run as test
  (mirleft/ocaml-tls#501, @hannesm - reported by @dinosaure)
avsm pushed a commit to avsm/opam-repository that referenced this pull request Sep 5, 2024
@hannesm @avsm
[new release] tls (6 packages) (1.0.0)
72f7fd3 
CHANGES:

* API breaking change: remove usage of Cstruct.t inside of TLS, use bytes
  and string instead (mirleft/ocaml-tls#497 by @art-w, @hannesm, @dinosaure, @reynir)
  Performance is up to 3x improved (bandwidth), 2x improvement for handshake/s
  on an Intel Core(TM) i7-5600U CPU @ 2.60GHz
* FEATURE: add tls-miou-unix package, which adds miou support for TLS
  (mirleft/ocaml-tls#494 mirleft/ocaml-tls#503 @dinosaure)
* FEATURE: tls-lwt and tls-async: allow TLS over an existing connection
  `Tls_lwt.client_of_channels : Tls.Config.client -> ?host:[`host] Domain_name.t -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  and
  `Tls_lwt.server_of_channels : Tls.Config.server -> Lwt_io.input_channel * Lwt_io.output_channel -> t Lwt.t`
  (mirleft/ocaml-tls#499 @art-w @MisterDA)
* API breaking changes: revise errors - reduce the polymorphic variant
  in size, align it with RFC specified errors, be in parts more precise
  about errors, in other parts skip data (mirleft/ocaml-tls#505, @hannesm - fixes mirleft/ocaml-tls#491)
  NB: if you relied on a specific error constructor, please open an issue
* Remove unused constructors from Packet.{alert_type, compression_methods,
  client_certificate_type, extension_type} (mirleft/ocaml-tls#505, @hannesm)
  NB: if you relied on specific constructors, please open an issue
* API breaking change: Tls.Config.{server,client} now return a result
  type instead of raising an exception (mirleft/ocaml-tls#502, @hannesm, fixes mirleft/ocaml-tls#411)
* FEATURE: add bench/speed.exe, a benchmark for bandwidth (for different
  ciphersuites) and handshakes (different key exchanges and private keys)
  (mirleft/ocaml-tls#500 @hannesm @dinosaure @reynir)
* BUGFIX: tests/feedback.exe update with TLS 1.3 semantics, run as test
  (mirleft/ocaml-tls#501, @hannesm - reported by @dinosaure)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@dinosaure dinosaure dinosaure approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hannesm @dinosaure