policy: Do not scan all entries when adding new ones#33313
Merged
jrajahalme merged 18 commits intocilium:mainfrom Jul 15, 2024
Merged
policy: Do not scan all entries when adding new ones#33313jrajahalme merged 18 commits intocilium:mainfrom
jrajahalme merged 18 commits intocilium:mainfrom
Conversation
Member
Author
|
/test |
0a6b651 to
36797fb
Compare
Member
Author
|
/test |
36797fb to
2ba67c8
Compare
Member
Author
|
/test |
ba7cfa3 to
dee008c
Compare
Member
Author
|
/test |
Member
Author
|
/test |
Add back the simple Go map to hold MapStateEntries, make trie just an index to it. This change allows adding additional indices in later commits. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Make mapstate modifiers private. This helps later when modifiers need to have selector cache locked. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Remove the initMap from newMapState() and add withState(), and the same for the interface returning versions (NewMapState() and WithState()). This helps reducing refactor changes from the next commit. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Copy the L4-key when creating new L3/L4 entries due to policy precedence rules and copy the Identity from the L3-only key. This way the code is simpler and less error prone. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the type mapStateMap directly rather than via mapState when adding l3l4 entries for Auth policies. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Add CIDR index to mapStateMap alongside the ID map. This allows for more efficient lookup of super/sub-set entries for deny rule insertion. This requires passing Identities interface along so that the CIDR can be gotten from the selector cache. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use PortProto and CIDR tries instead of scanning all entries when preserving precedence for deny rules. On a test benchmark with 1000 random allow and deny CIDR rules each, this speeds up MapState generation by >1800x, hopefully making deny policies useful in practical situations. Optional validator interface is added to mapState so that we can keep the existing identityIsSupersetOf logic in unit tests as additional validation that the ancestor and descendats logic is visiting only expeted entries. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
The added L3/L4 entries were created with 0 proxy port, while the proxy port should be copied from the L4 rule. Auth type should propagate from a L3 rule with protocol and wildcard port when another L3-wildcard rule without auth type has the same protocol and a specific port. In this case a new entry with the L3 and proto from the L3-rule and the port (and the same protocol) from the L4-only rule should be created. Add unit testing to validate the above. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the port/proto trie index to exclude uninteresing keys from the scan needed for propagating auth properties from more general rules to more specific ones. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the cidr index to find interesting items for visibility key injection without ranging over all entries in the MapState. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Currently there are no users so it is a good time to remove these. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
fe0fa25 to
5691e79
Compare
Member
Author
|
Removed empty commit due to merged #33449 |
Member
Author
|
/test |
tommyp1ckles
approved these changes
Jul 12, 2024
Contributor
tommyp1ckles
left a comment
There was a problem hiding this comment.
Approved for @cilium/endpoint
markpash
approved these changes
Jul 15, 2024
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Jul 22, 2024
Check Key.TrafficDirection for invalid values before using it to index an array. Return from denyPreferredInsertWithChanges() without doing anything if the given key has an invalid traffic direction. Dump a stack trace into logs if this happens so that we can trace where the invalid key originates from. Fixes: cilium#33313 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jul 31, 2024
Check Key.TrafficDirection for invalid values before using it to index an array. Return from denyPreferredInsertWithChanges() without doing anything if the given key has an invalid traffic direction. Dump a stack trace into logs if this happens so that we can trace where the invalid key originates from. Fixes: #33313 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hold MapStateEntries in
entriesmap,trieandcidrwithin thetrieare additional indices for the same MapStateEntries.cidronly indexes keys with CIDR identities, whiletrieindexes keys by traffic direction, protocol, destination port and mask.Use the
trieand the newcidrindex when adding entries to MapState with deny entries, and the existingtrieindex when adding entries to MapState with auth entries. Likewise, use thetrieindex when adding visibility entries. With this we no longer scan through all entries in MapState when adding new entries.MapState now contains an optional validator interface that can be set in unit tests to add calls to validation code for the new ancestor/descendants code. This is currently enabled only for the
denyPreferredInsertWithChanges.On local benchmarking for generating MapState for an egress policy with 1000 deny and 1000 allow CIDR rules:
Before:
After:
This is a 1820x speedup :-)
Using the PortProto
triein addition to the newcidrtrie as suggested by @nathanjsweet made this 8x faster for this specific test case.