Skip to content

Commit e235fea

Browse files
tlrxtlrx
committed
Adjust for 7.17.1
1 parent ec8bc67 commit e235fea

4 files changed

Lines changed: 33 additions & 93 deletions

File tree

‎docs/reference/modules/indices/recovery.asciidoc‎

Lines changed: 19 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,11 @@ the resources available to handle the extra load that will result.
115115

116116
NOTE: {cloud-only}
117117

118+
WARNING: This feature is available in {es} 7.17.1 and 8.0.1 onwards but is not
119+
supported in {es} 8.0.0. As such the recovery settings for managed services should
120+
be removed before upgrading to 8.0.0. It is possible to configure the settings in
121+
7.17.1 and then upgrade to 8.0.1 directly.
122+
118123
When running {es} as a managed service, the following settings allow the
119124
service to specify absolute maximum bandwidths for disk reads, disk writes, and
120125
network traffic on each node, and permit you to control the maximum recovery
@@ -136,10 +141,8 @@ writes, and network transfers using `node.bandwidth.recovery.disk.read`,
136141
`node.bandwidth.recovery.disk.write` and `node.bandwidth.recovery.network`
137142
respectively, and you can set the proportion of the absolute maximum bandwidth
138143
that may be used for recoveries by adjusting
139-
`node.bandwidth.recovery.factor.read` and
140-
`node.bandwidth.recovery.factor.write`. If the {operator-feature} is enabled
141-
then the service may also set default proportions using operator-only variants
142-
of these settings.
144+
`node.bandwidth.recovery.operator.factor.read` and
145+
`node.bandwidth.recovery.operator.factor.write`.
143146

144147
If you set `indices.recovery.max_bytes_per_sec` then {es} will use its value
145148
for the maximum recovery bandwidth, as long as this does not exceed the
@@ -172,51 +175,27 @@ for a recovery-like workload on the node, which applies to both reads and
172175
writes. If set, `node.bandwidth.recovery.disk.read` and
173176
`node.bandwidth.recovery.disk.write` must also be set.
174177

175-
`node.bandwidth.recovery.factor.read`::
176-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the maximum read
177-
bandwidth that may be used for recoveries if
178-
`indices.recovery.max_bytes_per_sec` is not set. Must be greater than `0` and
179-
not greater than `1`. If not set, the value of
180-
`node.bandwidth.recovery.operator.factor.read` is used. If no factor settings
181-
are set then the value `0.4` is used.
182-
183-
`node.bandwidth.recovery.factor.write`::
184-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the maximum
185-
write bandwidth that may be used for recoveries if
186-
`indices.recovery.max_bytes_per_sec` is not set. Must be greater than `0` and
187-
not greater than `1`. If not set, the value of
188-
`node.bandwidth.recovery.operator.factor.write` is used. If no factor settings
189-
are set then the value `0.4` is used.
190-
191178
`node.bandwidth.recovery.operator.factor.read`::
192-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the maximum read
193-
bandwidth that may be used for recoveries if
194-
`indices.recovery.max_bytes_per_sec` and `node.bandwidth.recovery.factor.read`
195-
are not set. Must be greater than `0` and not greater than `1`. If not set, the
179+
(float) The proportion of the maximum read
180+
bandwidth that may be used for recoveries if `indices.recovery.max_bytes_per_sec`
181+
is not set. Must be greater than `0` and not greater than `1`. If not set, the
196182
value of `node.bandwidth.recovery.operator.factor` is used. If no factor
197-
settings are set then the value `0.4` is used. When the {operator-feature} is
198-
enabled, this setting can be updated only by operator users.
183+
settings are set then the value `0.4` is used.
199184

200185
`node.bandwidth.recovery.operator.factor.write`::
201-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the maximum
202-
write bandwidth that may be used for recoveries if
203-
`indices.recovery.max_bytes_per_sec` and `node.bandwidth.recovery.factor.write`
204-
are not set. Must be greater than `0` and not greater than `1`. If not set, the
186+
(float) The proportion of the maximum
187+
write bandwidth that may be used for recoveries if `indices.recovery.max_bytes_per_sec`
188+
is not set. Must be greater than `0` and not greater than `1`. If not set, the
205189
value of `node.bandwidth.recovery.operator.factor` is used. If no factor
206-
settings are set then the value `0.4` is used. When the {operator-feature} is
207-
enabled, this setting can be updated only by operator users.
190+
settings are set then the value `0.4` is used.
208191

209192
`node.bandwidth.recovery.operator.factor`::
210-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the maximum
193+
(float) The proportion of the maximum
211194
bandwidth that may be used for recoveries if neither
212195
`indices.recovery.max_bytes_per_sec` nor any other factor settings are set.
213-
Must be greater than `0` and not greater than `1`. Defaults to `0.4`. When the
214-
{operator-feature} is enabled, this setting can be updated only by operator
215-
users.
196+
Must be greater than `0` and not greater than `1`. Defaults to `0.4`.
216197

217198
`node.bandwidth.recovery.operator.factor.max_overcommit`::
218-
(float, <<dynamic-cluster-setting,dynamic>>) The proportion of the absolute
199+
(float) The proportion of the absolute
219200
maximum bandwidth that may be used for recoveries regardless of any other
220-
settings. Must be greater than `0`. Defaults to `100`. When the
221-
{operator-feature} is enabled, this setting can be updated only by operator
222-
users.
201+
settings. Must be greater than `0`. Defaults to `100`.

‎server/src/main/java/org/elasticsearch/common/settings/ClusterSettings.java‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -237,8 +237,6 @@ public void apply(Settings value, Settings current, Settings previous) {
237237
RecoverySettings.INDICES_RECOVERY_USE_SNAPSHOTS_SETTING,
238238
RecoverySettings.INDICES_RECOVERY_MAX_CONCURRENT_SNAPSHOT_FILE_DOWNLOADS,
239239
RecoverySettings.INDICES_RECOVERY_MAX_CONCURRENT_SNAPSHOT_FILE_DOWNLOADS_PER_NODE,
240-
RecoverySettings.NODE_BANDWIDTH_RECOVERY_FACTOR_READ_SETTING,
241-
RecoverySettings.NODE_BANDWIDTH_RECOVERY_FACTOR_WRITE_SETTING,
242240
RecoverySettings.NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_SETTING,
243241
RecoverySettings.NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_READ_SETTING,
244242
RecoverySettings.NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_WRITE_SETTING,

‎server/src/main/java/org/elasticsearch/indices/recovery/RecoverySettings.java‎

Lines changed: 8 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,6 @@
4141
import java.util.stream.Collectors;
4242

4343
import static org.elasticsearch.common.settings.Setting.parseInt;
44-
import static org.elasticsearch.node.NodeRoleSettings.NODE_ROLES_SETTING;
4544

4645
public class RecoverySettings {
4746
public static final Version SNAPSHOT_RECOVERIES_SUPPORTED_VERSION = Version.V_7_15_0;
@@ -117,18 +116,7 @@ public class RecoverySettings {
117116
100d, // high default overcommit
118117
1d,
119118
Double.MAX_VALUE,
120-
Property.NodeScope,
121-
Property.OperatorDynamic
122-
);
123-
124-
public static final Setting<Double> NODE_BANDWIDTH_RECOVERY_FACTOR_WRITE_SETTING = factorSetting(
125-
"node.bandwidth.recovery.factor.write",
126-
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_WRITE_SETTING
127-
);
128-
129-
public static final Setting<Double> NODE_BANDWIDTH_RECOVERY_FACTOR_READ_SETTING = factorSetting(
130-
"node.bandwidth.recovery.factor.read",
131-
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_READ_SETTING
119+
Property.NodeScope
132120
);
133121

134122
static final List<Setting<?>> NODE_BANDWIDTH_RECOVERY_SETTINGS = Arrays.asList(
@@ -180,26 +168,15 @@ private static Setting<Double> operatorFactorSetting(String key, double defaultV
180168
if (v == 0d) {
181169
throw new IllegalArgumentException("Failed to validate value [" + v + "] for factor setting [" + key + "] must be > [0]");
182170
}
183-
}, Property.NodeScope, Property.OperatorDynamic);
171+
}, Property.NodeScope);
184172
}
185173

186174
private static Setting<Double> operatorFactorSetting(String key) {
187175
return new Setting<>(key, NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_SETTING, s -> Setting.parseDouble(s, 0d, 1d, key), v -> {
188176
if (v == 0d) {
189177
throw new IllegalArgumentException("Failed to validate value [" + v + "] for factor setting [" + key + "] must be > [0]");
190178
}
191-
}, Property.NodeScope, Property.OperatorDynamic);
192-
}
193-
194-
/**
195-
* User-defined factors have a value in (0.0, 1.0] and fall back to a corresponding operator factor setting.
196-
*/
197-
private static Setting<Double> factorSetting(String key, Setting<Double> operatorFallback) {
198-
return new Setting<>(key, operatorFallback, s -> Setting.parseDouble(s, 0d, 1d, key), v -> {
199-
if (v == 0d) {
200-
throw new IllegalArgumentException("Failed to validate value [" + v + "] for factor setting [" + key + "] must be > [0]");
201-
}
202-
}, Property.NodeScope, Property.Dynamic);
179+
}, Property.NodeScope);
203180
}
204181

205182
static final ByteSizeValue DEFAULT_MAX_BYTES_PER_SEC = new ByteSizeValue(40L, ByteSizeUnit.MB);
@@ -443,17 +420,14 @@ public RecoverySettings(Settings settings, ClusterSettings clusterSettings) {
443420
this::computeMaxBytesPerSec,
444421
Arrays.asList(
445422
INDICES_RECOVERY_MAX_BYTES_PER_SEC_SETTING,
446-
NODE_BANDWIDTH_RECOVERY_FACTOR_READ_SETTING,
447-
NODE_BANDWIDTH_RECOVERY_FACTOR_WRITE_SETTING,
423+
// non dynamic settings but they are used to update max bytes per sec
448424
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_SETTING,
449425
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_READ_SETTING,
450426
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_WRITE_SETTING,
451427
NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_MAX_OVERCOMMIT_SETTING,
452-
// non dynamic settings but they are used to update max bytes per sec
453428
NODE_BANDWIDTH_RECOVERY_DISK_WRITE_SETTING,
454429
NODE_BANDWIDTH_RECOVERY_DISK_READ_SETTING,
455-
NODE_BANDWIDTH_RECOVERY_NETWORK_SETTING,
456-
NODE_ROLES_SETTING
430+
NODE_BANDWIDTH_RECOVERY_NETWORK_SETTING
457431
)
458432
);
459433
}
@@ -483,7 +457,7 @@ public RecoverySettings(Settings settings, ClusterSettings clusterSettings) {
483457
}
484458

485459
private void computeMaxBytesPerSec(Settings settings) {
486-
// limit as computed before 8.1.0
460+
// limit as computed before node bandwidth recovery settings were introduced
487461
final long defaultBytesPerSec = Math.max(INDICES_RECOVERY_MAX_BYTES_PER_SEC_SETTING.get(settings).getBytes(), 0L);
488462

489463
// available network bandwidth
@@ -492,7 +466,7 @@ private void computeMaxBytesPerSec(Settings settings) {
492466
// read bandwidth
493467
final long readBytesPerSec;
494468
if (availableDiskReadBandwidth.getBytes() > 0L && networkBandwidthBytesPerSec > 0L) {
495-
double readFactor = NODE_BANDWIDTH_RECOVERY_FACTOR_READ_SETTING.get(settings);
469+
double readFactor = NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_READ_SETTING.get(settings);
496470
readBytesPerSec = Math.round(Math.min(availableDiskReadBandwidth.getBytes(), networkBandwidthBytesPerSec) * readFactor);
497471
} else {
498472
readBytesPerSec = 0L;
@@ -501,7 +475,7 @@ private void computeMaxBytesPerSec(Settings settings) {
501475
// write bandwidth
502476
final long writeBytesPerSec;
503477
if (availableDiskWriteBandwidth.getBytes() > 0L && networkBandwidthBytesPerSec > 0L) {
504-
double writeFactor = NODE_BANDWIDTH_RECOVERY_FACTOR_WRITE_SETTING.get(settings);
478+
double writeFactor = NODE_BANDWIDTH_RECOVERY_OPERATOR_FACTOR_WRITE_SETTING.get(settings);
505479
writeBytesPerSec = Math.round(Math.min(availableDiskWriteBandwidth.getBytes(), networkBandwidthBytesPerSec) * writeFactor);
506480
} else {
507481
writeBytesPerSec = 0L;

‎server/src/test/java/org/elasticsearch/indices/recovery/RecoverySettingsTests.java‎

Lines changed: 6 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -231,17 +231,13 @@ public void testMaxBytesPerSecOnDataNodeWithIndicesRecoveryMaxBytesPerSecAndOver
231231
.withRandomMemory()
232232
.build()
233233
.getMaxBytesPerSec(),
234-
equalTo(
235-
ByteSizeValue.ofBytes(
236-
Math.round(requireNonNullElse(maxOvercommitFactor, 100.d) * ByteSizeValue.ofMb(250).getBytes())
237-
)
238-
)
234+
equalTo(ByteSizeValue.ofBytes(Math.round(requireNonNullElse(maxOvercommitFactor, 100.d) * ByteSizeValue.ofMb(250).getBytes())))
239235
);
240236
}
241237

242238
public void testMaxBytesPerSecOnDataNodeWithAvailableBandwidths() {
243239
assertThat(
244-
"Data node should use pre 8.1.0 default because available bandwidths are lower",
240+
"Data node should use default because available bandwidths are lower",
245241
nodeRecoverySettings().withRoles(randomDataNodeRoles())
246242
.withRandomMemory()
247243
.withNetworkBandwidth(ByteSizeValue.ofGb(between(1, 10)))
@@ -279,9 +275,7 @@ public void testMaxBytesPerSecOnDataNodeWithAvailableBandwidths() {
279275
.withOperatorDefaultFactor(factor)
280276
.build()
281277
.getMaxBytesPerSec(),
282-
equalTo(
283-
ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * networkBandwidth.getBytes()))
284-
)
278+
equalTo(ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * networkBandwidth.getBytes())))
285279
);
286280

287281
final ByteSizeValue diskReadBandwidth = ByteSizeValue.ofMb(randomFrom(100, 250));
@@ -295,9 +289,7 @@ public void testMaxBytesPerSecOnDataNodeWithAvailableBandwidths() {
295289
.withOperatorDefaultFactor(factor)
296290
.build()
297291
.getMaxBytesPerSec(),
298-
equalTo(
299-
ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * diskReadBandwidth.getBytes()))
300-
)
292+
equalTo(ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * diskReadBandwidth.getBytes())))
301293
);
302294

303295
final ByteSizeValue diskWriteBandwidth = ByteSizeValue.ofMb(randomFrom(100, 250));
@@ -311,9 +303,7 @@ public void testMaxBytesPerSecOnDataNodeWithAvailableBandwidths() {
311303
.withOperatorDefaultFactor(factor)
312304
.build()
313305
.getMaxBytesPerSec(),
314-
equalTo(
315-
ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * diskWriteBandwidth.getBytes()))
316-
)
306+
equalTo(ByteSizeValue.ofBytes(Math.round(requireNonNullElse(factor, DEFAULT_FACTOR_VALUE) * diskWriteBandwidth.getBytes())))
317307
);
318308
}
319309

@@ -421,8 +411,7 @@ private static Set<String> randomDataNodeRoles() {
421411
if (randomBoolean()) {
422412
roles.addAll(
423413
randomSubsetOf(
424-
DiscoveryNodeRole.BUILT_IN_ROLES
425-
.stream()
414+
DiscoveryNodeRole.BUILT_IN_ROLES.stream()
426415
.filter(role -> role.canContainData() == false)
427416
.map(DiscoveryNodeRole::roleName)
428417
.collect(Collectors.toSet())

0 commit comments

Comments
 (0)