hyperscan: remove package and replace it with vectorscan#27382
Merged
hnyman merged 5 commits intoopenwrt:masterfrom Sep 20, 2025
Merged
hyperscan: remove package and replace it with vectorscan#27382hnyman merged 5 commits intoopenwrt:masterfrom
hnyman merged 5 commits intoopenwrt:masterfrom
Conversation
Contributor
Author
|
accidentally closed |
17247d8 to
28a7fa8
Compare
9daf2c4 to
b27d387
Compare
4 tasks
Vectorscan is fork of Hyperscan, a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API. Currently ARM NEON/ASIMD and Power VSX are 100% functional. ARM SVE2 support is in ongoing with access to hardware now. More platforms will follow in the future. The performance difference of snort3 compiled against this is sizable for aarch64 confirmed on two different SoCs: Test SoC #1 flogic/glinet_gl-mt6000 IDS mode: Download speed wo/ vectorscan: 91.2 ±0.21 Mbit/s (n=3) Download speed using vectorscan: 331.0 ±27.34 Mbit/s (n=3) Gain of 3.6x IPS mode: Download speed wo/ vectorscan: 30.0 ±0.06 Mbit/s (n=3) Download speed using vectorscan: 52.9 ±0.78 Mbit/s (n=3) Gain of 1.8x Notes: * Data generated on snapshot build on 12-Apr-2024 using kernel 6.6.26, snort 3.1.84.0, vectorscan 5.4.11. * Speedtest script hitting the same server. * Snort rules file of was 37,917 lines/22 MB. * In all cases, single core CPU saturation occurred which speaks to the efficiency gains supplied by vectorscan. Test Soc #2 bcm2712/RPi5B IPS mode: Download speed wo/ vectorscan: 164.3 ±0.64 Mbit/s (n=3) Download speed using vectorscan: 232.8 ±0.26 Mbit/s (n=3) Gain of 1.4x Notes: * Data generated on snapshot build on 13-Apr-2024 using kernel 6.1.86, snort 3.1.84.0, vectorscan 5.4.11. * Google fiber speedtest (https://fiber.google.com/speedtest/) hitting the same server. * Snort rules contained 39,801 rules/22 MB. * In all cases, single core CPU saturation occurred which speaks to the efficiency gains supplied by vectorscan. Build system: x86/64 Build-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc Run-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc (Intel N150 based box) Co-authored-by: Tianling Shen <cnsztl@gmail.com> Co-authored-by: Jeffery To <jeffery.to@gmail.com> Signed-off-by: John Audia <therealgraysky@proton.me>
Drop 100-remove-HAVE_HS_COMPILE_LIT-to-work-around-upstream-b.patch as it was only needed to fix the build against hyperscan. Vectorscan builds fine without it. Signed-off-by: John Audia <therealgraysky@proton.me>
Remove hyperscan since Intel announced a proprietary/closed source license beginning with hyperscan 5.5[1,2] and a general lock of support for the 5.4.x branch which has not seen a commit since 19-Apr-2023[3]. 1. https://networkbuilders.intel.com/docs/networkbuilders/accelerate-snort-performance-with-hyperscan-and-intel-xeon-processors-on-public-clouds-1680176363.pdf 2. https://www.phoronix.com/news/Intel-Hyperscan-Now-Proprietary (and references therein) 3. intel/hyperscan@bc3b191 Signed-off-by: John Audia <therealgraysky@proton.me>
* Replacement of hyperscan-runtime reference with vectorscan-runtime * Added support for all aarch64 targets which I believe is exhaustive For x86 and x86/64, I found that vectorscan is truly a drop-in replacement for hyperscan as assessed by speedtests with snort3 running on my Intel N150 PC. CPU load during the test with each condition was nearly saturating on a single core for both cases on a symmetrical Gbps line. Using: https://www.waveform.com/tools/bufferbloat in IPS mode: Download speed w/ hyperscan: 950-960 Mbit/s (n=2) Download speed w/ vectorscan: 942-960 Mbit/s (n=2) Using: https://www.speedtest.net in IPS mode: Download speed w/ hyperscan: 996-1002 Mbit/s (n=2) Download speed w/ vectorscan: 993-988 Mbit/s (n=2) Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc (Intel N150 based box running snort3) Signed-off-by: John Audia <therealgraysky@proton.me>
When snort is run with the --version option, it advertises components' versions in the output. Add a patch to modify the output to clearly show vectorscan is in use. Signed-off-by: John Audia <therealgraysky@proton.me>
Contributor
Author
|
@hnyman - what do you think about the merge-ability of this PR? |
Contributor
|
PR looks straightforward. (I do not use snort, so no first-hand experience.) |
Contributor
|
Looks fine. |
Contributor
|
Thank you, @graysky2! |
Member
|
I looked at it very very briefly. I don't even take a look at any statistics number: Because vectorscan does not compile for i386 - VectorCamp/vectorscan#343 (comment) While looking at Ubuntu and Debian repositories: Ubuntu:
Debian:
So, I would say that hyperscan should work for i386 that leads to question - should we have hyperscan for i386? 🤔 🤔 |
Contributor
Author
|
We could add back hyperscan for i386. I don't see a problem with that. For question a. the answer should be everyone given the sizable performance boost. For b I don't know. I have often wondered if openwrt could create a tool similar to Arch Linux's pkgstats. You don't need 100% of the users reporting data. Probably something much smaller. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📦 Package Details
Maintainer: me
(You can find this by checking the history of the package
Makefile.)Description:
Remove hyperscan since Intel announced a proprietary/closed source license beginning with hyperscan 5.5 and a general lock of support for the 5.4.x branch which has not seen a commit since 19-Apr-2023.
Add vectorscan, a drop-in replacement with support for targets beyond x86 and x86/64. See commit message for details and for tests on two aarch64 boxes that show substantial performance gains running snort3.
Update our snort3 package to use vectorscan-runtime (I am the co-maintainer of snort3 and @flyn-org is the other).
🧪 Run Testing Details
✅ Formalities
If your PR contains a patch:
git am(e.g., subject line, commit description, etc.)
We must try to upstream patches to reduce maintenance burden.