Skip to content

core: support chunked transfer for image files#10820

Merged
DaanHoogland merged 3 commits intoapache:4.20from
shapeblue:fix-checked-img-dwnld
May 12, 2025
Merged

core: support chunked transfer for image files#10820
DaanHoogland merged 3 commits intoapache:4.20from
shapeblue:fix-checked-img-dwnld

Conversation

@shwstppr
Copy link
Copy Markdown
Contributor

@shwstppr shwstppr commented May 6, 2025 •
edited
Loading

Description

Fixes #10658
Fixes download of with chunked transfer encoding for non-directdownload templates and ISOs

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested download of:

SSVM logs - tail -f /var/log/cloud.log | grep -E "HttpTemplateDownloader|DownloadManagerImpl" 
2025-05-11T12:53:29,498 INFO  [storage.template.HttpTemplateDownloader] (AgentRequest-Handler-7:[]) No credentials configured for host=10.1.3.130:80
2025-05-11T12:53:29,503 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Remote size for http://10.1.3.130/cks/setup-1.26.0.iso found to be (660.03 MB) 692094976
2025-05-11T12:53:29,505 INFO  [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Starting download from http://10.1.3.130/cks/setup-1.26.0.iso to /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ remoteSize=(660.03 MB) 692094976 , max size=(50.00 GB) 53687091200
2025-05-11T12:53:29,528 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Verified format of downloading file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ is supported
2025-05-11T12:53:31,223 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-2:[]) Reached expected remote size limit: 692094976 bytes
2025-05-11T12:53:31,241 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Download Completion for jobId: 94db7f97-1dd0-4df0-923c-4d86c4c2c208, status=DOWNLOAD_FINISHED
2025-05-11T12:53:31,242 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) local: /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_, bytes=(660.03 MB) 692094976, error=Downloaded (660.03 MB) 692094976 bytes (download complete remote=unknown bytes), pct=100
2025-05-11T12:53:31,242 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) No checksum available for 'dnld16851435041626252415tmp_'
2025-05-11T12:53:35,477 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) computed checksum: {SHA-512}ad026d55e0fda7dcb65764660258084f9985d8800b684228a367cf7f90a44d05fb3ff83d9d4e6bc0da2b7049c8bdbfa8c432b342faf96386f92666bcbec8a216
2025-05-11T12:53:35,482 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Executing command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d non-chunked -h -n 210-2-66ee1468-58ee-388e-bf9f-9c8aca93b27d.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ -u ].
2025-05-11T12:53:35,596 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Successfully executed process [10257] for command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d non-chunked -h -n 210-2-66ee1468-58ee-388e-bf9f-9c8aca93b27d.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_ -u ].
2025-05-11T12:53:35,681 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Cleaning-up temporary download file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210/dnld16851435041626252415tmp_
2025-05-11T12:53:35,684 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-2:[]) Deleting directory /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/210, if empty, as part of cleanup
2025-05-11T12:54:16,591 INFO  [storage.template.HttpTemplateDownloader] (AgentRequest-Handler-10:[]) No credentials configured for host=releases.rancher.com:443
2025-05-11T12:54:17,039 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso is using chunked transfer encoding, possible remote size: 883715325
2025-05-11T12:54:17,043 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Remote size for https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso found to be (842.78 MB) 883715325
2025-05-11T12:54:17,051 INFO  [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Starting download from https://releases.rancher.com/harvester/v1.3.2/harvester-v1.3.2-amd64-net-install.iso to /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ remoteSize=(842.78 MB) 883715325 , max size=(50.00 GB) 53687091200
2025-05-11T12:54:17,533 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Verified format of downloading file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ is supported
2025-05-11T12:54:45,608 DEBUG [storage.template.HttpTemplateDownloader] (pool-2-thread-3:[]) Reached expected remote size limit: 883715325 bytes
2025-05-11T12:54:46,113 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Download Completion for jobId: 472028ff-d07e-4b67-abb3-d5065a87d401, status=DOWNLOAD_FINISHED
2025-05-11T12:54:46,114 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) local: /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_, bytes=(842.78 MB) 883722310, error=Downloaded (842.78 MB) 883722310 bytes (download complete remote=unknown bytes), pct=100
2025-05-11T12:54:46,115 INFO  [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) No checksum available for 'dnld6098655088962065044tmp_'
2025-05-11T12:54:52,088 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) computed checksum: {SHA-512}44fb1821bbaef863c68da65d84c0363a86efe24592f6b2eff64957b163f1d700041ccde856af418c0a9cc369a1188cb79d44dcaf665fafdebe17e7906742db2d
2025-05-11T12:54:52,092 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Executing command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d chunked -h -n 211-2-21936ca5-db4d-3030-8f63-ed051b48865a.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ -u ].
2025-05-11T12:54:52,212 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Successfully executed process [10285] for command [/usr/local/cloud/systemvm/scripts/storage/secondary/createtmplt.sh -s 2 -S 53687091200 -d chunked -h -n 211-2-21936ca5-db4d-3030-8f63-ed051b48865a.iso -t /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211 -f /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_ -u ].
2025-05-11T12:54:52,263 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Cleaning-up temporary download file /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211/dnld6098655088962065044tmp_
2025-05-11T12:54:52,265 DEBUG [storage.template.DownloadManagerImpl] (pool-2-thread-3:[]) Deleting directory /mnt/SecStorage/037ced3e-bc30-388e-a0c2-d307b6f8ffb1/template/tmpl/2/211, if empty, as part of cleanup

How did you try to break this feature and the system with this change?

@shwstppr
core: support chunked transfer for image files
3d404fd 
Fixes apache#10658

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
@codecov
Copy link
Copy Markdown

codecov bot commented May 6, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 49 lines in your changes missing coverage. Please review.

Project coverage is 16.13%. Comparing base (f6d0590) to head (0b6185a).
Report is 17 commits behind head on 4.20.

Files with missing lines Patch % Lines
...cloud/storage/template/HttpTemplateDownloader.java 0.00% 49 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.20   #10820      +/-   ##
============================================
- Coverage     16.13%   16.13%   -0.01%     
  Complexity    13220    13220              
============================================
  Files          5651     5651              
  Lines        496740   496785      +45     
  Branches      60183    60187       +4     
============================================
+ Hits          80148    80150       +2     
- Misses       407674   407710      +36     
- Partials       8918     8925       +7
Flag Coverage Δ
uitests 4.00% <ø> (+<0.01%) ⬆️
unittests 16.98% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland DaanHoogland mentioned this pull request May 6, 2025
Closed
@DaanHoogland
Copy link
Copy Markdown
Contributor

DaanHoogland commented May 6, 2025

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 6, 2025

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 6, 2025

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13273

@shwstppr
fix
7989d58 
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
@DaanHoogland
Copy link
Copy Markdown
Contributor

DaanHoogland commented May 6, 2025

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 6, 2025

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 6, 2025

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13275

@DaanHoogland
Copy link
Copy Markdown
Contributor

DaanHoogland commented May 7, 2025

@blueorangutan test

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 7, 2025

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 7, 2025

[SF] Trillian test result (tid-13224)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53033 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10820-t13224-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@shwstppr
fix
0b6185a 
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
@shwstppr
Copy link
Copy Markdown
Contributor Author

shwstppr commented May 11, 2025

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 11, 2025

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 11, 2025

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13329

@shwstppr
Copy link
Copy Markdown
Contributor Author

shwstppr commented May 11, 2025

@blueorangutan test

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 11, 2025

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented May 12, 2025

[SF] Trillian test result (tid-13258)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53285 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10820-t13258-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@DaanHoogland
Copy link
Copy Markdown
Contributor

DaanHoogland commented May 12, 2025

user reported positive result (see #10658 (comment))

@shwstppr shwstppr marked this pull request as ready for review May 12, 2025 09:59
@DaanHoogland DaanHoogland merged commit 265f4cd into apache:4.20 May 12, 2025
21 of 26 checks passed
@DaanHoogland DaanHoogland deleted the fix-checked-img-dwnld branch May 12, 2025 10:50
@DaanHoogland DaanHoogland linked an issue May 12, 2025 that may be closed by this pull request
Downloads of ISOs fails when the server is using "chunked" transfers #10658
Closed
dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 19, 2025
@shwstppr @dhslove
core: support chunked transfer for image files (apache#10820)
65a5766
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DaanHoogland DaanHoogland DaanHoogland approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Downloads of ISOs fails when the server is using "chunked" transfers

3 participants

@shwstppr @DaanHoogland @blueorangutan