fix(field-mask): Allow FieldMasking on oneof variants#43924
fix(field-mask): Allow FieldMasking on oneof variants#43924julianwiedmann merged 2 commits intocilium:mainfrom
Conversation
f615425 to
6ffc875
Compare
|
Commit ae47b24 does not match "(?m)^Signed-off-by:". Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin |
|
@devodev Hey! Could you have a look at this? I ran in to an awkward scenario while testing! :) |
6027796 to
3eef569
Compare
devodev
left a comment
There was a problem hiding this comment.
Overall this looks great, and awesome catch BTW!
Especially if we can make it before the v1.19.0 release 💪
I left some comments but nothing major.
One thing I may ask is to update your commit messages to describe the issue at hand and the actual fix.
The PR description already explains very well the issue regarding fieldmask. As for the timestamp, I would explain why timestamping at Add time vs Export time matters and also why the field should not be part of the aggregationKey (essentially that we always want to have the time field in aggregated events).
Lastly we should update the release-notes to be brief but descriptive of what we fixed and why instead of the actual change made. What I mean is its not clear to me as a lambda user of Hubble export what "Allow FieldMasking on oneof variants" means. I would also try and include context such as "Hubble export aggregation" so its easier to grok than simply "aggregation" or "aggregated flows".
Thank you again for the contribution :)
7dee7af to
219151f
Compare
|
/test |
|
@mereta I'm not sure the commit messages are descriptive enough. If I don't have the PR in front of me, I would need to analyze the added tests to figure out what was the issue and how it was fixed exactly. Could you explain in the body of the commit what was the issue observed and how it was fixed (the actual change to the fieldmask copy logic). Similarly, I'd like to see more meat for "aggregation: Enrich aggregated flows with timestamp for temporal ctx". Also make sure the title provides enough context. I would prefix both with |
Oh sorry, yes i can add that I misunderstood. I'll replace the fix() & chore() with 'hubble' as the commit message is limited to 75 chars. :) |
….TCP) When multiple oneof variants are specified in a field mask (e.g., both TCP and UDP), the previous implementation would copy all variants, creating spurious structures for inactive ones. This caused identical flows to generate different aggregation keys. Added oneof-aware copying using WhichOneof() to only copy the active variant, preventing spurious structures and ensuring consistent keys. Signed-off-by: mereta <mereta.degutyte@hotmail.co.uk>
…ontext Aggregation only keeps and aggregates on the fields specified in the fieldAggregate. There is no way to preserve a timestamp, and specifying time in fieldAggregate defeats aggregation. The solution is to preserve the 1st occurring timestamp in the processedFlow after the aggregation key is generated. The aggregation logic is not affected and temporal context is preserved. Signed-off-by: mereta <mereta.degutyte@hotmail.co.uk>
|
/test |
Problem
While testing field aggregation with
source.namespace,destination.namespace, and L4/L7 protocol fields, I discovered that flows were not aggregating correctly. When a field mask included multiple oneof variants (e.g.,l4.TCP.destination_portandl4.UDP.destination_port), the field mask Copy() method would create spurious empty structures for ALL specified variants, not just the active one.This caused TCP flows to incorrectly include empty UDP structures in their aggregation keys, preventing proper aggregation even when the actual flow data was identical.
Solution
1st commit:
Modified
FieldMask.Copy()to detect oneof fields using protobuf reflection (fd.ContainingOneof()) and only copy the active variant by checkingWhichOneof(). This ensures that only the actual protocol present in the source flow is copied to the destination, preventing spurious structures.This enables users to specify fine-grained oneof variant fields (e.g.,
l4.TCP.destination_port,l4.UDP.destination_port) in field masks for flow aggregation and export without creating incorrect aggregation keys.Addition in 2nd commit:
Enrich Aggregated hubble flow logs with timestamp to have temporal context.
Aggregation only keeps and aggregates on the fields specified in the
fieldAggregate.There is no way to preserve a timestamp, and specifying time in
fieldAggretatedefeats aggregation.The solution is to preserve the 1st occurring timestamp in the
processedFlowafter the aggregation key is generated.The aggregation logic is not affected and temporal context is preserved.
Testing
fieldmask_test.godemonstrating the fix works for L4 (TCP/UDP/SCTP/ICMP) and L7 (HTTP/DNS) oneof variantsaggregator_test.goshowing TCP+HTTP flows now correctly aggregate when the field mask includes both TCP/UDP and HTTP/DNS pathsManual Testing Steps:
Manual Verrification:
BYO CNI Cilium Cluster
helm upgrade cilium cilium/cilium --version 1.19-devcilium-flowlog-configConfigMapvar/run/cilium/hubbleBoth UDP and TCP are logged:

FieldMask Changes:
New ability to mask nested oneof fields.
No other changes in FM functionality.