Skip to content

2322 scitag source destination #2359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
amadio merged 3 commits into from
Oct 30, 2024
Merged

2322 scitag source destination #2359

amadio merged 3 commits into from
Oct 30, 2024

Conversation

@ccaffy
Copy link
Contributor

@ccaffy ccaffy commented Oct 17, 2024

Includes @abh3 PMark implementation adaptation + HTTP + HTTP-TPC PMark modification

abh3 and others added 3 commits October 17, 2024 05:23
@abh3
[PMark] Readjust the definition of src and dest relative to the server.
7f02a63
@ccaffy
[XrdHttp] Added packet marking appname for firefly emission hint
c6f707f 
In the new SciTag specification, the content of the emitted firefly
should always be created from the data sender point of view.
For HTTP PUT: the srcIp is the client, dstIP is the server,
sentBytes=socketReceivedBytes (~=fileSize),
receivedBytes=socketSentBytes (~=0)
For HTTP GET: the srcIP is the server, dstIP is the client,
sentBytes=socketSentBytes(~=fileSize)
receivedBytes=socketReceivedBytes(~=0)

The distinction is therefore done via the pmark.appname CGI that is
equal to "http-get" if the client request is a GET request or equal to
"http-put" if not.
@ccaffy
[XrdTpcTPC] PMark - Adapted packet marking to fit the new specifications
02ea3d5
@ccaffy ccaffy requested review from abh3 and amadio October 17, 2024 15:28
@ccaffy
Copy link
Contributor Author

ccaffy commented Oct 18, 2024

Here are the results of the tests.

HTTP TPC PULL:
curl -v --capath /etc/grid-security/certificates -L -X COPY -H "source: $DST" -H "SciTag: 144" "$SRC"

Active server emits the firefly, Source port is passive server, destinationPort is activeServer, sent~=fileSize, received~=0

<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"start","current-
time":"2024-10-18T08:48:24.201540+00:00","start-time":"2024-10-18T08:48:24.201540+00:00"},"usage":
{"received":518,"sent":0},"netlink":{"rtt":0.023},"context":{"experiment-id":2,"activity-id":16,"application":"http-put"},"flow-
id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":2001,"dst-port":49378}}
<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"end","current-
time":"2024-10-18T08:48:24.230146+00:00","start-time":"2024-10-18T08:48:24.201540+00:00","end-
time":"2024-10-18T08:48:24.230146+00:00"},"usage":{"received":813,"sent":10507049},"netlink":{"rtt":0.053},"context":
{"experiment-id":2,"activity-id":16,"application":"http-put"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-
ip":"REDACTED","protocol":"tcp","src-port":2001,"dst-port":49378}}

HTTP TPC PULL with TransferHeaderSciTag set (the passive server will emit the firefly):
SourcePort is passive server, destinationPort is active server, sent~=fileSize, received~=0

<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"start","current-
time":"2024-10-18T08:55:11.051306+00:00","start-time":"2024-10-18T08:55:11.051306+00:00"},"usage":
{"received":801,"sent":7055},"netlink":{"rtt":0.020},"context":{"experiment-id":2,"activity-id":16,"application":"http-
get"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":2001,"dst-
port":40126}}
<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":
{"state":"end","current-time":"2024-10-18T08:55:11.068430+00:00","start-time":"2024-10-18T08:55:11.051306+00:00","end-
time":"2024-10-18T08:55:11.068430+00:00"},"usage":{"received":825,"sent":10507024},"netlink":{"rtt":0.877},"context":{"experiment-id":2,"activity-id":16,"application":"http-get"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":2001,"dst-port":40126}}

HTTP TPC PUSH:
curl -v --capath /etc/grid-security/certificates -L -X COPY -H "destination: $DST" -H "SciTag: 144" "$SRC"
Active server emits the firefly. Source port is activeServer, destinationPort is passive, sent~=fileSize, received~=0

<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"start","current-
time":"2024-10-18T08:51:55.179441+00:00","start-time":"2024-10-18T08:51:55.179441+00:00"},"usage":
{"received":0,"sent":188},"netlink":{"rtt":0.021},"context":{"experiment-id":2,"activity-id":16,"application":"http-get"},"flow-
id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":54326,"dst-port":2001}}
<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"end","current-
time":"2024-10-18T08:51:55.189122+00:00","start-time":"2024-10-18T08:51:55.179441+00:00","end-
time":"2024-10-18T08:51:55.189122+00:00"},"usage":{"received":185,"sent":10485948},"netlink":{"rtt":0.398},"context":
{"experiment-id":2,"activity-id":16,"application":"http-get"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-
ip":"REDACTED","protocol":"tcp","src-port":54326,"dst-port":2001}}

HTTP PUT:
Firefly emitted by the server, sourcePort is client, destinationPort is server, sent=fileSize, received~=0

<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"start","current-
time":"2024-10-18T08:57:36.630223+00:00","start-time":"2024-10-18T08:57:36.630223+00:00"},"usage":
{"received":7055,"sent":828},"netlink":{"rtt":0.031},"context":{"experiment-id":2,"activity-id":16,"application":"http-
put"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":38048,"dst-
port":1096}}
<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"end","current-
time":"2024-10-18T08:57:36.681023+00:00","start-time":"2024-10-18T08:57:36.630223+00:00","end-
time":"2024-10-18T08:57:36.681023+00:00"},"usage":{"received":7306,"sent":10500692},"netlink":{"rtt":0.065},"context":
{"experiment-id":2,"activity-id":16,"application":"http-put"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-
ip":"REDACTED","protocol":"tcp","src-port":38048,"dst-port":1096}}

HTTP GET:
Firefly emitted by the server, sourcePort is server, destinationPort is client, sent~=fileSize, received~=0

<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"start","current-
time":"2024-10-18T08:58:52.702923+00:00","start-time":"2024-10-18T08:58:52.702923+00:00"},"usage":
{"received":779,"sent":6976},"netlink":{"rtt":0.020},"context":{"experiment-id":2,"activity-id":16,"application":"http-
get"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-ip":"REDACTED","protocol":"tcp","src-port":1096,"dst-
port":49300}}
<134>1 - xrootd-server.cern.ch xrootd - firefly-json - {"version":1,"flow-lifecycle":{"state":"end","current-
time":"2024-10-18T08:58:52.710155+00:00","start-time":"2024-10-18T08:58:52.702923+00:00","end-
time":"2024-10-18T08:58:52.710155+00:00"},"usage":{"received":803,"sent":1057167},"netlink":{"rtt":0.062},"context":
{"experiment-id":2,"activity-id":16,"application":"http-get"},"flow-id":{"afi":"ipv4","src-ip":"REDACTED","dst-
ip":"REDACTED","protocol":"tcp","src-port":1096,"dst-port":49300}}

All good!

@abh3 abh3 mentioned this pull request Oct 24, 2024
Closed
abh3
abh3 approved these changes Oct 24, 2024
View reviewed changes
Copy link
Member

@abh3 abh3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've already tested this and it's OK. However, we need @amadio to figure out why Alpine build fails before we fully approve this.

@abh3 abh3 mentioned this pull request Oct 24, 2024
Closed
@amadio
Copy link
Member

amadio commented Oct 25, 2024

This is fine to be merged, the Alpine problem is due to git issues on GitHub's side. I will rebase and push again to run the CI on Alpine, then merge. You don't need to do anything. Cheers,

@amadio amadio linked an issue Oct 25, 2024 that may be closed by this pull request
PMark: swap source and destination #2322
Closed
@amadio amadio merged commit b0f089a into xrootd:devel Oct 30, 2024
@amadio amadio added this to the 5.8.0 milestone Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abh3 abh3 abh3 approved these changes

@amadio amadio Awaiting requested review from amadio

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

5.8.0

Development

Successfully merging this pull request may close these issues.

PMark: swap source and destination

3 participants

@ccaffy @amadio @abh3