Skip to content

Tpc worker pool #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dynamic-entropy wants to merge 11 commits into from
Closed

Tpc worker pool #3

dynamic-entropy wants to merge 11 commits into from
+859 −263

Conversation

@dynamic-entropy
Copy link
Owner

@dynamic-entropy dynamic-entropy commented Jun 2, 2025

No description provided.

@dynamic-entropy dynamic-entropy force-pushed the tpc-worker-pool branch 3 times, most recently from 60692cd to 346858c Compare June 3, 2025 08:24
@dynamic-entropy dynamic-entropy force-pushed the tpc-worker-pool branch 6 times, most recently from ab8a77b to fde8a00 Compare June 30, 2025 12:29
dynamic-entropy added a commit that referenced this pull request Jul 1, 2025
@dynamic-entropy
Fix memory leak detected by Address sanitizer
29af271 
- buff is duplicated into XrdOucStream and never freed

130: ==77706==ERROR: LeakSanitizer: detected memory leaks
130:
130: Direct leak of 25 byte(s) in 1 object(s) allocated from:
130:     #0 0xffff866de3e0 in strdup (/lib64/libasan.so.8+0xde3e0) (BuildId: 8a06fba3c3703d6d4fc1517732173e023d50b811)
130:     #1 0x40258c in main /xrootd/src/XrdApps/XrdAppsCconfig.cc:189
130:     #2 0xffff85c560d8 in __libc_start_call_main (/lib64/libc.so.6+0x260d8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
130:     #3 0xffff85c561b8 in __libc_start_main_alias_1 (/lib64/libc.so.6+0x261b8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
130:     #4 0x402d6c in _start (/home/xrootd/build/bin/cconfig+0x402d6c) (BuildId: abdf333ad86ac01dbfaab35089b0f2ca46c00ff1)
dynamic-entropy added a commit that referenced this pull request Jul 1, 2025
@dynamic-entropy
Fix memory leak detected by Address sanitizer
cfc3538 
- buff is duplicated into XrdOucStream and never freed

130: ==77706==ERROR: LeakSanitizer: detected memory leaks
130:
130: Direct leak of 25 byte(s) in 1 object(s) allocated from:
130:     #0 0xffff866de3e0 in strdup (/lib64/libasan.so.8+0xde3e0) (BuildId: 8a06fba3c3703d6d4fc1517732173e023d50b811)
130:     #1 0x40258c in main /xrootd/src/XrdApps/XrdAppsCconfig.cc:189
130:     #2 0xffff85c560d8 in __libc_start_call_main (/lib64/libc.so.6+0x260d8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
130:     #3 0xffff85c561b8 in __libc_start_main_alias_1 (/lib64/libc.so.6+0x261b8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
130:     #4 0x402d6c in _start (/home/xrootd/build/bin/cconfig+0x402d6c) (BuildId: abdf333ad86ac01dbfaab35089b0f2ca46c00ff1)
dynamic-entropy added a commit that referenced this pull request Jul 1, 2025
@dynamic-entropy
Fix memory leak in scitokens test reported by address sanitizer
8904ba3 
OSSL_PARAM_get_octet_string() retrieves an OCTET string from the parameter pointed to by p. The OCTETs are either stored into *val with a length limit of max_len or, in the case when *val is NULL, memory is allocated and max_len is ignored. *used_len is populated with the number of OCTETs stored. If val is NULL then the OCTETS are not stored, but *used_len is still populated. If memory is allocated by this function, it must be freed by the caller.

```
110: ==71089==ERROR: LeakSanitizer: detected memory leaks
110:
110: Direct leak of 65 byte(s) in 1 object(s) allocated from:
110:     #0 0xffff940e3fa0 in malloc (/lib64/libasan.so.8+0xe3fa0) (BuildId: 8a06fba3c3703d6d4fc1517732173e023d50b811)
110:     #1 0xffff93b70cc8 in CRYPTO_malloc (/lib64/libcrypto.so.3+0x170cc8) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #2 0xffff93b734d0 in get_string_internal (/lib64/libcrypto.so.3+0x1734d0) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #3 0xffff93b73544 in OSSL_PARAM_get_octet_string (/lib64/libcrypto.so.3+0x173544) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #4 0x403268 in readPubkey(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /xrootd/tests/scitokens/XrdScitokensCreateJwks.cc:116
110:     #5 0x401f8c in main /xrootd/tests/scitokens/XrdScitokensCreateJwks.cc:180
110:     #6 0xffff934560d8 in __libc_start_call_main (/lib64/libc.so.6+0x260d8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
110:     #7 0xffff934561b8 in __libc_start_main_alias_1 (/lib64/libc.so.6+0x261b8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
110:     xrootd#8 0x4022ec in _start (/home/xrootd/build/bin/xrdscitokens-create-jwks+0x4022ec) (BuildId: 59427405cd76ed61fc5a88686a7466e1ec3ecc68)
110:
```
dynamic-entropy added a commit that referenced this pull request Jul 1, 2025
@dynamic-entropy @amadio
Fix memory leak in scitokens test reported by address sanitizer
defb87c 
OSSL_PARAM_get_octet_string() retrieves an OCTET string from the parameter pointed to by p. The OCTETs are either stored into *val with a length limit of max_len or, in the case when *val is NULL, memory is allocated and max_len is ignored. *used_len is populated with the number of OCTETs stored. If val is NULL then the OCTETS are not stored, but *used_len is still populated. If memory is allocated by this function, it must be freed by the caller.

```
110: ==71089==ERROR: LeakSanitizer: detected memory leaks
110:
110: Direct leak of 65 byte(s) in 1 object(s) allocated from:
110:     #0 0xffff940e3fa0 in malloc (/lib64/libasan.so.8+0xe3fa0) (BuildId: 8a06fba3c3703d6d4fc1517732173e023d50b811)
110:     #1 0xffff93b70cc8 in CRYPTO_malloc (/lib64/libcrypto.so.3+0x170cc8) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #2 0xffff93b734d0 in get_string_internal (/lib64/libcrypto.so.3+0x1734d0) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #3 0xffff93b73544 in OSSL_PARAM_get_octet_string (/lib64/libcrypto.so.3+0x173544) (BuildId: 33e0997b4260371d5963783548a106ff6f0d6fc2)
110:     #4 0x403268 in readPubkey(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /xrootd/tests/scitokens/XrdScitokensCreateJwks.cc:116
110:     #5 0x401f8c in main /xrootd/tests/scitokens/XrdScitokensCreateJwks.cc:180
110:     #6 0xffff934560d8 in __libc_start_call_main (/lib64/libc.so.6+0x260d8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
110:     #7 0xffff934561b8 in __libc_start_main_alias_1 (/lib64/libc.so.6+0x261b8) (BuildId: 6b190cb37e36bb79643c3b0727477dee387d4cf8)
110:     xrootd#8 0x4022ec in _start (/home/xrootd/build/bin/xrdscitokens-create-jwks+0x4022ec) (BuildId: 59427405cd76ed61fc5a88686a7466e1ec3ecc68)
110:
```
@dynamic-entropy dynamic-entropy force-pushed the tpc-worker-pool branch 7 times, most recently from 0c15429 to 46f0657 Compare July 30, 2025 15:16
@dynamic-entropy dynamic-entropy force-pushed the tpc-worker-pool branch 8 times, most recently from d10da37 to bf7e22d Compare August 12, 2025 11:25
dynamic-entropy and others added 11 commits October 16, 2025 12:17
@dynamic-entropy @amadio
Updated perform_http_tpc to correctly detect
12876f3 
transfer success or failure by checking the last line
of the chunked reponse, since the HTTP 201 code
is always returned regardless of actual status.

Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Add scitag flow configuration to test
4836023 
Add scitag flow query string in tests

Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Add test for cancellations
2d44b4b 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Add multistream transfers to the test
5d3b1e4 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@bbockelm @dynamic-entropy @amadio
[XrdHttpTpc] Framework for worker pool for TPC requests
9400e68 
Co-authored-by: Rahul Chauhan <rahul.chauhan@cern.ch>
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Change ownership of curl socket callbacks
86c038c 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Refactor PMarkManager and TPCRequestManager to add SciTa…
e9f00c4 
…g integration

Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Implement cancellation in TPC Pool
4c5183f 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Update remote connection list from curl
8710b04 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Make max_transfers and max_pending requests configurable
725e374 
Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
@dynamic-entropy @amadio
[XrdHttpTpc] Use vorg as part of the queue identifier as well as work…
5d9e92a 
…er and request labels

Also fix nomenclature of identifier and labels:
A queue has a identifier (unique)
Workers and Requets have labels (not uniquet) that map to a queue

Signed-off-by: Rahul Chauhan <rahul.chauhan@cern.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dynamic-entropy @bbockelm