Skip to content

[Issue 5059][offloaders] Delete offloaded with offloadpolicies #6354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
KannarFr wants to merge 1 commit into from
Closed

[Issue 5059][offloaders] Delete offloaded with offloadpolicies #6354

KannarFr wants to merge 1 commit into from

Conversation

@KannarFr
Copy link
Contributor

@KannarFr KannarFr commented Feb 18, 2020 •
edited
Loading

Master Issue: #5059

Motivation

Describe deletion max lag and threshold bytes to remove data from offloaded tiered storage.

Modifications

Will update once it's finished.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • The public API: yes
  • The default values of configurations: yes (introducing new one)
  • The rest endpoints: yes (modifying/adding)
  • The admin cli options: yes (modifying/adding)

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? not documented
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation. Will do

@KannarFr KannarFr force-pushed the delete-offloaded-with-offloadpolicies branch from c39eebc to f7a7b0d Compare February 18, 2020 14:04
@KannarFr
Copy link
Contributor Author

KannarFr commented Feb 18, 2020

Oh, jcloud already contains deleteOffloaded :).

@sijie sijie added this to the 2.6.0 milestone Feb 18, 2020
@sijie
Copy link
Member

sijie commented Feb 18, 2020

loop in @gaoran10 to help review this as well.

@KannarFr
Copy link
Contributor Author

KannarFr commented Feb 18, 2020 via email

@KannarFr
Copy link
Contributor Author

KannarFr commented Feb 19, 2020

So #6283 is not done, I thought it was. I need it for this PR.

@KannarFr KannarFr force-pushed the delete-offloaded-with-offloadpolicies branch from ae5f0ea to 476d112 Compare February 19, 2020 23:21
@KannarFr KannarFr changed the title [Issue 5059][offloaders] WIP - Delete offloaded with offloadpolicies [Issue 5059][offloaders] Delete offloaded with offloadpolicies Feb 24, 2020
@KannarFr KannarFr mentioned this pull request Feb 26, 2020
Merged
1 task
sijie pushed a commit that referenced this pull request Mar 28, 2020
@KannarFr
[Issue 6283][tiered-storage] Offload policies per namespace (#6422)
347d385 
Fixes #6283 

### Modifications

Define and use custom deletionLag and threshold for offloadpolicies per ns.
All is stuff is required for #6354.
@KannarFr KannarFr force-pushed the delete-offloaded-with-offloadpolicies branch 6 times, most recently from 58bbd81 to 8dc4d6e Compare March 30, 2020 06:54
@KannarFr KannarFr force-pushed the delete-offloaded-with-offloadpolicies branch from 8dc4d6e to 89afddb Compare March 31, 2020 03:22
@KannarFr
Copy link
Contributor Author

KannarFr commented Mar 31, 2020

Just realized that this if useless because it's retention jobs T_T.

@KannarFr KannarFr closed this Mar 31, 2020
jiazhai pushed a commit that referenced this pull request May 8, 2020
@KannarFr @jiazhai
[Issue 6283][tiered-storage] Offload policies per namespace (#6422)
c3fa923 
Fixes #6283

### Modifications

Define and use custom deletionLag and threshold for offloadpolicies per ns.
All is stuff is required for #6354.

(cherry picked from commit 347d385)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
@KannarFr @huangdx0726
[Issue 6283][tiered-storage] Offload policies per namespace (apache#6422
5c2ece3 
)

Fixes apache#6283 

### Modifications

Define and use custom deletionLag and threshold for offloadpolicies per ns.
All is stuff is required for apache#6354.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sijie sijie Awaiting requested review from sijie

@jiazhai jiazhai Awaiting requested review from jiazhai

@ivankelly ivankelly Awaiting requested review from ivankelly

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

Assignees

@KannarFr KannarFr

Labels

area/tieredstorage

Projects

None yet

Milestone

2.6.0

Development

Successfully merging this pull request may close these issues.

2 participants

@KannarFr @sijie