Motivation

Background

• The oldest behaviour of offload settings: Pulsar only supports setting Broker-level offload policies and namespace-level thresholds. and handle namespace-level thresholds by the following fields:
  • {@link Policies#offload_deletion_lag_ms}
  • {@link Policies#offload_threshold}
  • {@link Policies#offload_threshold_in_seconds}
• After Namespace level offloader #6183, Pulsar supports namespace-level policies, which were supported by {@link Policies#offload_policies}. And the thresholds were moved to the following fields:
  • {@link Policies#offload_policies} -> {@link OffloadPoliciesImpl#getManagedLedgerOffloadDeletionLagInMillis}
  • {@link Policies#offload_policies} -> {@link OffloadPoliciesImpl#getManagedLedgerOffloadThresholdInBytes}
  • {@link Policies#offload_policies} -> {@link OffloadPoliciesImpl#getManagedLedgerOffloadThresholdInSeconds}
• [Issue 6283][tiered-storage] Offload policies per namespace #6422 made setting offload thresholds({@link Policies#offload_deletion_lag_ms}, {@link Policies#offload_threshold}, {@link Policies#offload_threshold_in_seconds}) affect namespace policies({@link Policies#offload_policies} )
• But neither mechanisms work well together

The issue we encountered, Thanks @horizonzy for investigating the issue

> pulsar-admin topicPolicies get-offload-policies <topic>
null

> more conf/broker.conf | grep -i "managedLedgerOffload"
managedLedgerOffloadDeletionLagMs=63072000000
managedLedgerOffloadAutoTriggerSizeThresholdBytes=-1
managedLedgerOffloadThresholdInSeconds=-1

> pulsar-admin namespaces get-offload-threshold <namespace>
offloadThresholdInBytes: 0
offloadThresholdInSeconds: -1

> pulsar-admin topicPolicies get-offload-policies <topic> -ap
{
  ...
  "managedLedgerOffloadThresholdInSeconds" : -1,
  "managedLedgerOffloadThresholdInBytes" : -1,
}

• How to reproduce the issue
  • create namespace-level offload policies
  • set namespace-level offload policies thresholds
  • remove namespace-level offload policies
  • get topic-level offload policies with -ap, you will get a dirty view

Modifications

To make the offload policies compatible with the old policies, use the old policies' threshold({@link Policies#offload_deletion_lag_ms}, {@link Policies#offload_threshold}, {@link Policies#offload_threshold_in_seconds}) if the new policies({@link Policies#offload_policies} ) are not set. Once the new fields are set, the old fields will be removed.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x