Skip to content

policy: Do not scan all entries when adding new ones#33313

Merged
jrajahalme merged 18 commits intocilium:mainfrom
jrajahalme:policy-index-by-cidr
Jul 15, 2024
Merged

policy: Do not scan all entries when adding new ones#33313
jrajahalme merged 18 commits intocilium:mainfrom
jrajahalme:policy-index-by-cidr

Conversation

@jrajahalme
Copy link
Copy Markdown
Member

@jrajahalme jrajahalme commented Jun 21, 2024

Hold MapStateEntries in entries map, trie and cidr within the trie are additional indices for the same MapStateEntries. cidr only indexes keys with CIDR identities, while trie indexes keys by traffic direction, protocol, destination port and mask.

Use the trie and the new cidr index when adding entries to MapState with deny entries, and the existing trie index when adding entries to MapState with auth entries. Likewise, use the trie index when adding visibility entries. With this we no longer scan through all entries in MapState when adding new entries.

MapState now contains an optional validator interface that can be set in unit tests to add calls to validation code for the new ancestor/descendants code. This is currently enabled only for the denyPreferredInsertWithChanges.

On local benchmarking for generating MapState for an egress policy with 1000 deny and 1000 allow CIDR rules:

Before:

Number of MapState entries: 117515
goos: linux
goarch: arm64
pkg: github.com/cilium/cilium/pkg/policy
BenchmarkRegenerateCIDRDenyPolicyRules-12    	       1	763361909494 ns/op
PASS
ok  	github.com/cilium/cilium/pkg/policy	763.696s

After:

Number of MapState entries: 117515
goos: linux
goarch: arm64
pkg: github.com/cilium/cilium/pkg/policy
BenchmarkRegenerateCIDRDenyPolicyRules-12    	Number of MapState entries: 117515
Number of MapState entries: 117515
       3	 418981451 ns/op
PASS
ok  	github.com/cilium/cilium/pkg/policy	3.434s

This is a 1820x speedup :-)

Using the PortProto trie in addition to the new cidr trie as suggested by @nathanjsweet made this 8x faster for this specific test case.

Deny rule processing has been improved to run faster, especially on larger policies.

@jrajahalme jrajahalme added kind/enhancement This would improve or streamline existing functionality. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. labels Jun 21, 2024
@jrajahalme jrajahalme requested review from a team as code owners June 21, 2024 10:04
@jrajahalme jrajahalme requested a review from doniacld June 21, 2024 10:04
@maintainer-s-little-helper maintainer-s-little-helper bot added dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Jun 21, 2024
@jrajahalme jrajahalme marked this pull request as draft June 21, 2024 10:04
@jrajahalme
Copy link
Copy Markdown
Member Author

/test

@jrajahalme jrajahalme force-pushed the policy-index-by-cidr branch from 0a6b651 to 36797fb Compare June 21, 2024 16:54
@jrajahalme
Copy link
Copy Markdown
Member Author

/test

@jrajahalme jrajahalme added the release-note/minor This PR changes functionality that users may find relevant to operating Cilium. label Jun 21, 2024
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jun 21, 2024
@jrajahalme jrajahalme changed the title policy: Improve drop rule handling by ~100x policy: Improve deny rule handling by ~100x Jun 21, 2024
@jrajahalme jrajahalme force-pushed the policy-index-by-cidr branch from 36797fb to 2ba67c8 Compare June 21, 2024 21:50
@jrajahalme
Copy link
Copy Markdown
Member Author

/test

@jrajahalme jrajahalme force-pushed the policy-index-by-cidr branch 2 times, most recently from ba7cfa3 to dee008c Compare June 22, 2024 08:02
@jrajahalme jrajahalme marked this pull request as ready for review June 22, 2024 08:02
@jrajahalme jrajahalme requested review from lmb, nathanjsweet and squeed and removed request for doniacld June 22, 2024 08:04
@jrajahalme jrajahalme changed the title policy: Improve deny rule handling by ~100x policy: Improve deny rule handling Jun 22, 2024
@jrajahalme
Copy link
Copy Markdown
Member Author

/test

@jrajahalme
Copy link
Copy Markdown
Member Author

/test

Add back the simple Go map to hold MapStateEntries, make trie just an
index to it.

This change allows adding additional indices in later commits.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Make mapstate modifiers private. This helps later when modifiers need to
have selector cache locked.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Remove the initMap from newMapState() and add withState(), and the same
for the interface returning versions (NewMapState() and WithState()). This
helps reducing refactor changes from the next commit.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Copy the L4-key when creating new L3/L4 entries due to policy precedence
rules and copy the Identity from the L3-only key. This way the code is
simpler and less error prone.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the type mapStateMap directly rather than via mapState when adding
l3l4 entries for Auth policies.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Add CIDR index to mapStateMap alongside the ID map. This allows for more
efficient lookup of super/sub-set entries for deny rule insertion.

This requires passing Identities interface along so that the CIDR can be
gotten from the selector cache.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use PortProto and CIDR tries instead of scanning all entries when
preserving precedence for deny rules.

On a test benchmark with 1000 random allow and deny CIDR rules each, this
speeds up MapState generation by >1800x, hopefully making deny policies
useful in practical situations.

Optional validator interface is added to mapState so that we can keep the
existing identityIsSupersetOf logic in unit tests as additional
validation that the ancestor and descendats logic is visiting only
expeted entries.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
The added L3/L4 entries were created with 0 proxy port, while the proxy
port should be copied from the L4 rule.

Auth type should propagate from a L3 rule with protocol and wildcard port
when another L3-wildcard rule without auth type has the same protocol and
a specific port. In this case a new entry with the L3 and proto from the
L3-rule and the port (and the same protocol) from the L4-only rule should
be created.

Add unit testing to validate the above.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the port/proto trie index to exclude uninteresing keys from the scan
needed for propagating auth properties from more general rules to more
specific ones.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Use the cidr index to find interesting items for visibility key injection
without ranging over all entries in the MapState.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Currently there are no users so it is a good time to remove these.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
@jrajahalme jrajahalme force-pushed the policy-index-by-cidr branch from fe0fa25 to 5691e79 Compare July 11, 2024 14:35
@jrajahalme
Copy link
Copy Markdown
Member Author

Removed empty commit due to merged #33449

@jrajahalme
Copy link
Copy Markdown
Member Author

/test

Copy link
Copy Markdown
Contributor

@tommyp1ckles tommyp1ckles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for @cilium/endpoint

@jrajahalme jrajahalme removed the request for review from squeed July 15, 2024 20:32
@jrajahalme jrajahalme added this pull request to the merge queue Jul 15, 2024
Merged via the queue into cilium:main with commit fb47ea5 Jul 15, 2024
@jrajahalme jrajahalme deleted the policy-index-by-cidr branch July 15, 2024 20:42
@maintainer-s-little-helper maintainer-s-little-helper bot added ready-to-merge This PR has passed all tests and received consensus from code owners to merge. labels Jul 15, 2024
jrajahalme added a commit to jrajahalme/cilium that referenced this pull request Jul 22, 2024
Check Key.TrafficDirection for invalid values before using it to index an
array. Return from denyPreferredInsertWithChanges() without doing
anything if the given key has an invalid traffic direction. Dump a stack
trace into logs if this happens so that we can trace where the invalid
key originates from.

Fixes: cilium#33313

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
github-merge-queue bot pushed a commit that referenced this pull request Jul 31, 2024
Check Key.TrafficDirection for invalid values before using it to index an
array. Return from denyPreferredInsertWithChanges() without doing
anything if the given key has an invalid traffic direction. Dump a stack
trace into logs if this happens so that we can trace where the invalid
key originates from.

Fixes: #33313

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement This would improve or streamline existing functionality. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants