Skip to content

Commit 9ee0743

Browse files
karenyrxkarenyrx
committed
Merge remote-tracking branch 'ossupstream/main' into test-pass-reference
2 parents 3df4626 + 1a40b69 commit 9ee0743

165 files changed

Lines changed: 3038 additions & 520 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/benchmark-configs.json‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,5 +187,22 @@
187187
"data_instance_config": "4vCPU, 32G Mem, 16G Heap"
188188
},
189189
"baseline_cluster_config": "x64-r5.xlarge-single-node-1-shard-0-replica-baseline"
190+
},
191+
"id_12": {
192+
"description": "Search only test-procedure for HTTP_LOGS, uses snapshot to restore the data for OS-3.0.0",
193+
"supported_major_versions": ["3"],
194+
"cluster-benchmark-configs": {
195+
"SINGLE_NODE_CLUSTER": "true",
196+
"MIN_DISTRIBUTION": "true",
197+
"TEST_WORKLOAD": "http_logs",
198+
"WORKLOAD_PARAMS": "{\"snapshot_repo_name\":\"benchmark-workloads-repo-3x\",\"snapshot_bucket_name\":\"benchmark-workload-snapshots\",\"snapshot_region\":\"us-east-1\",\"snapshot_base_path\":\"workload-snapshots-3x\",\"snapshot_name\":\"http_logs_1_shard\"}",
199+
"CAPTURE_NODE_STAT": "true",
200+
"TEST_PROCEDURE": "restore-from-snapshot"
201+
},
202+
"cluster_configuration": {
203+
"size": "Single-Node",
204+
"data_instance_config": "4vCPU, 32G Mem, 16G Heap"
205+
},
206+
"baseline_cluster_config": "x64-r5.xlarge-1-shard-0-replica-snapshot-baseline"
190207
}
191-
}
208+
}

‎.github/workflows/wrapper.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,4 @@ jobs:
88
runs-on: ubuntu-latest
99
steps:
1010
- uses: actions/checkout@v4
11-
- uses: gradle/wrapper-validation-action@v3
11+
- uses: gradle/actions/wrapper-validation@v3

‎CHANGELOG.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
77
### Added
88
- Add support for linux riscv64 platform ([#18156](https://github.com/opensearch-project/OpenSearch/pull/18156))
99
- [Rule based auto-tagging] Add get rule API ([#17336](https://github.com/opensearch-project/OpenSearch/pull/17336))
10+
- [Rule based auto-tagging] Add Delete Rule API ([#18184](https://github.com/opensearch-project/OpenSearch/pull/18184))
1011
- Implement parallel shard refresh behind cluster settings ([#17782](https://github.com/opensearch-project/OpenSearch/pull/17782))
1112
- Bump OpenSearch Core main branch to 3.0.0 ([#18039](https://github.com/opensearch-project/OpenSearch/pull/18039))
1213
- Update API of Message in index to add the timestamp for lag calculation in ingestion polling ([#17977](https://github.com/opensearch-project/OpenSearch/pull/17977/))
@@ -25,28 +26,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
2526
- Decouple the init of Crypto Plugin and KeyProvider in CryptoRegistry ([18270](https://github.com/opensearch-project/OpenSearch/pull18270)))
2627
- Support cluster write block in pull-based ingestion ([#18280](https://github.com/opensearch-project/OpenSearch/pull/18280)))
2728
- Use QueryCoordinatorContext for the rewrite in validate API. ([#18272](https://github.com/opensearch-project/OpenSearch/pull/18272))
29+
- Upgrade crypto kms plugin dependencies for AWS SDK v2.x. ([#18268](https://github.com/opensearch-project/OpenSearch/pull/18268))
30+
- Add support for `matched_fields` with the unified highlighter ([#18164](https://github.com/opensearch-project/OpenSearch/issues/18164))
31+
- [repository-s3] Add support for SSE-KMS and S3 bucket owner verification ([#18312](https://github.com/opensearch-project/OpenSearch/pull/18312))
2832
- Optimize gRPC perf by passing by reference ([#18303](https://github.com/opensearch-project/OpenSearch/pull/18303))
2933

3034
### Changed
35+
- Create generic DocRequest to better categorize ActionRequests ([#18269](https://github.com/opensearch-project/OpenSearch/pull/18269)))
3136

3237
### Dependencies
38+
- Update Apache Lucene from 10.1.0 to 10.2.1 ([#17961](https://github.com/opensearch-project/OpenSearch/pull/17961))
3339
- Bump `com.google.code.gson:gson` from 2.12.1 to 2.13.1 ([#17923](https://github.com/opensearch-project/OpenSearch/pull/17923), [#18266](https://github.com/opensearch-project/OpenSearch/pull/18266))
3440
- Bump `com.github.spotbugs:spotbugs-annotations` from 4.9.0 to 4.9.3 ([#17922](https://github.com/opensearch-project/OpenSearch/pull/17922))
3541
- Bump `com.microsoft.azure:msal4j` from 1.18.0 to 1.20.0 ([#17925](https://github.com/opensearch-project/OpenSearch/pull/17925))
3642
- Update Apache HttpClient5 and HttpCore5 (CVE-2025-27820) ([#18152](https://github.com/opensearch-project/OpenSearch/pull/18152))
3743
- Bump `org.apache.commons:commons-collections4` from 4.4 to 4.5.0 ([#18101](https://github.com/opensearch-project/OpenSearch/pull/18101))
3844
- Bump `netty` from 4.1.118.Final to 4.1.121.Final ([#18192](https://github.com/opensearch-project/OpenSearch/pull/18192))
3945
- Bump `org.apache.commons:commons-configuration2` from 2.11.0 to 2.12.0 ([#18103](https://github.com/opensearch-project/OpenSearch/pull/18103), [#18262](https://github.com/opensearch-project/OpenSearch/pull/18262))
40-
- Bump `com.nimbusds:nimbus-jose-jwt` from 10.0.2 to 10.2 ([#18104](https://github.com/opensearch-project/OpenSearch/pull/18104))
46+
- Bump `com.nimbusds:nimbus-jose-jwt` from 10.0.2 to 10.3 ([#18104](https://github.com/opensearch-project/OpenSearch/pull/18104), [#18336](https://github.com/opensearch-project/OpenSearch/pull/18336))
4147
- Bump `org.apache.commons:commons-text` from 1.13.0 to 1.13.1 ([#18102](https://github.com/opensearch-project/OpenSearch/pull/18102))
4248
- Bump `reactor-netty` from 1.2.4 to 1.2.5 ([#18243](https://github.com/opensearch-project/OpenSearch/pull/18243))
4349
- Bump `reactor` from 3.5.20 to 3.7.5 ([#18243](https://github.com/opensearch-project/OpenSearch/pull/18243))
4450
- Bump `com.azure:azure-core-http-netty` from 1.15.7 to 1.15.11 ([#18265](https://github.com/opensearch-project/OpenSearch/pull/18265))
4551
- Bump `lycheeverse/lychee-action` from 2.4.0 to 2.4.1 ([#18264](https://github.com/opensearch-project/OpenSearch/pull/18264))
52+
- Bump `com.maxmind.geoip2:geoip2` from 4.2.1 to 4.3.0 ([#18263](https://github.com/opensearch-project/OpenSearch/pull/18263))
53+
- Bump `com.azure:azure-json` from 1.3.0 to 1.5.0 ([#18335](https://github.com/opensearch-project/OpenSearch/pull/18335))
4654

4755
### Deprecated
4856

4957
### Removed
58+
- [repository-s3] Removed existing ineffective `server_side_encryption` setting ([#18312](https://github.com/opensearch-project/OpenSearch/pull/18312))
5059

5160
### Fixed
5261
- Fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop ([#17532](https://github.com/opensearch-project/OpenSearch/pull/17532))
@@ -56,6 +65,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5665
- Use Bad Request status for InputCoercionException ([#18161](https://github.com/opensearch-project/OpenSearch/pull/18161))
5766
- Null check field names in QueryStringQueryBuilder ([#18194](https://github.com/opensearch-project/OpenSearch/pull/18194))
5867
- Avoid NPE if on SnapshotInfo if 'shallow' boolean not present ([#18187](https://github.com/opensearch-project/OpenSearch/issues/18187))
68+
- Fix 'system call filter not installed' caused when network.host: 0.0.0.0 ([#18309](https://github.com/opensearch-project/OpenSearch/pull/18309))
69+
- Fix MatrixStatsAggregator reuse when mode parameter changes ([#18242](https://github.com/opensearch-project/OpenSearch/issues/18242))
5970

6071
### Security
6172

‎MAINTAINERS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
2626
| Owais Kazi | [owaiskazi19](https://github.com/owaiskazi19) | Amazon |
2727
| Pan Guixin | [bugmakerrrrrr](https://github.com/bugmakerrrrrr) | ByteDance |
2828
| Peter Nied | [peternied](https://github.com/peternied) | Amazon |
29+
| Rishabh Maurya | [rishabhmaurya](https://github.com/rishabhmaurya) | Amazon |
2930
| Rishikesh Pasham | [Rishikesh1159](https://github.com/Rishikesh1159) | Amazon |
3031
| Sachin Kale | [sachinpkale](https://github.com/sachinpkale) | Amazon |
3132
| Sarat Vemulapalli | [saratvemulapalli](https://github.com/saratvemulapalli) | Amazon |

‎distribution/packages/src/common/systemd/opensearch.service‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,8 @@ LockPersonality=yes
101101
# @ means allowed
102102
# ~ means not allowed
103103
# These syscalls are related to mmap which is needed for OpenSearch Services
104-
SystemCallFilter=madvise mincore mlock mlock2 munlock get_mempolicy sched_getaffinity sched_setaffinity fcntl
104+
SystemCallFilter=seccomp mincore
105+
SystemCallFilter=madvise mlock mlock2 munlock get_mempolicy sched_getaffinity sched_setaffinity fcntl
105106
SystemCallFilter=@system-service
106107
SystemCallFilter=~@reboot
107108
SystemCallFilter=~@swap

‎distribution/tools/keystore-cli/src/test/java/org/opensearch/tools/cli/keystore/KeyStoreWrapperTests.java‎

Lines changed: 0 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -61,12 +61,10 @@
6161
import java.io.EOFException;
6262
import java.io.IOException;
6363
import java.io.InputStream;
64-
import java.io.OutputStream;
6564
import java.nio.charset.StandardCharsets;
6665
import java.nio.file.FileSystem;
6766
import java.nio.file.Files;
6867
import java.nio.file.Path;
69-
import java.security.GeneralSecurityException;
7068
import java.security.KeyStore;
7169
import java.security.KeyStoreException;
7270
import java.security.MessageDigest;
@@ -413,31 +411,6 @@ public void testStringAndFileDistinction() throws Exception {
413411
assertThat(toByteArray(afterSave.getFile("file_setting")), equalTo("file_value".getBytes(StandardCharsets.UTF_8)));
414412
}
415413

416-
@AwaitsFix(bugUrl = "https://github.com/opensearch-project/OpenSearch/issues/468")
417-
public void testLegacyV3() throws GeneralSecurityException, IOException {
418-
final Path configDir = createTempDir();
419-
final Path keystore = configDir.resolve("opensearch.keystore");
420-
try (
421-
InputStream is = KeyStoreWrapperTests.class.getResourceAsStream("/format-v3-opensearch.keystore");
422-
OutputStream os = Files.newOutputStream(keystore)
423-
) {
424-
final byte[] buffer = new byte[4096];
425-
int readBytes;
426-
while ((readBytes = is.read(buffer)) > 0) {
427-
os.write(buffer, 0, readBytes);
428-
}
429-
}
430-
final KeyStoreWrapper wrapper = KeyStoreWrapper.load(configDir);
431-
assertNotNull(wrapper);
432-
wrapper.decrypt(new char[0]);
433-
assertThat(wrapper.getFormatVersion(), equalTo(3));
434-
assertThat(wrapper.getSettingNames(), equalTo(new HashSet<>(Arrays.asList("keystore.seed", "string_setting", "file_setting"))));
435-
assertThat(wrapper.getString("string_setting"), equalTo("string_value"));
436-
assertThat(toByteArray(wrapper.getFile("string_setting")), equalTo("string_value".getBytes(StandardCharsets.UTF_8)));
437-
assertThat(wrapper.getString("file_setting"), equalTo("file_value"));
438-
assertThat(toByteArray(wrapper.getFile("file_setting")), equalTo("file_value".getBytes(StandardCharsets.UTF_8)));
439-
}
440-
441414
private void generateV1() throws IOException, NoSuchAlgorithmException, NoSuchProviderException, CertificateException,
442415
InvalidKeySpecException, KeyStoreException {
443416
Path configDir = env.configDir();

‎distribution/tools/keystore-cli/src/test/java/org/opensearch/tools/cli/keystore/UpgradeKeyStoreCommandTests.java‎

Lines changed: 0 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -37,15 +37,9 @@
3737
import org.opensearch.common.settings.KeyStoreWrapper;
3838
import org.opensearch.env.Environment;
3939

40-
import java.io.InputStream;
41-
import java.io.OutputStream;
42-
import java.nio.file.Files;
43-
import java.nio.file.Path;
4440
import java.util.Map;
4541

4642
import static org.hamcrest.Matchers.containsString;
47-
import static org.hamcrest.Matchers.equalTo;
48-
import static org.hamcrest.Matchers.hasItem;
4943
import static org.hamcrest.Matchers.hasToString;
5044

5145
public class UpgradeKeyStoreCommandTests extends KeyStoreCommandTestCase {
@@ -62,32 +56,6 @@ protected Environment createEnv(final Map<String, String> settings) {
6256
};
6357
}
6458

65-
@AwaitsFix(bugUrl = "https://github.com/opensearch-project/OpenSearch/issues/468")
66-
public void testKeystoreUpgrade() throws Exception {
67-
final Path keystore = KeyStoreWrapper.keystorePath(env.configDir());
68-
try (
69-
InputStream is = KeyStoreWrapperTests.class.getResourceAsStream("/format-v3-opensearch.keystore");
70-
OutputStream os = Files.newOutputStream(keystore)
71-
) {
72-
final byte[] buffer = new byte[4096];
73-
int read;
74-
while ((read = is.read(buffer, 0, buffer.length)) >= 0) {
75-
os.write(buffer, 0, read);
76-
}
77-
}
78-
try (KeyStoreWrapper beforeUpgrade = KeyStoreWrapper.load(env.configDir())) {
79-
assertNotNull(beforeUpgrade);
80-
assertThat(beforeUpgrade.getFormatVersion(), equalTo(3));
81-
}
82-
execute();
83-
try (KeyStoreWrapper afterUpgrade = KeyStoreWrapper.load(env.configDir())) {
84-
assertNotNull(afterUpgrade);
85-
assertThat(afterUpgrade.getFormatVersion(), equalTo(KeyStoreWrapper.FORMAT_VERSION));
86-
afterUpgrade.decrypt(new char[0]);
87-
assertThat(afterUpgrade.getSettingNames(), hasItem(KeyStoreWrapper.SEED_SETTING.getKey()));
88-
}
89-
}
90-
9159
public void testKeystoreDoesNotExist() {
9260
final UserException e = expectThrows(UserException.class, this::execute);
9361
assertThat(e, hasToString(containsString("keystore not found at [" + KeyStoreWrapper.keystorePath(env.configDir()) + "]")));

‎distribution/tools/keystore-cli/src/test/resources/format-v3-elasticsearch.keystore‎

-281 Bytes
Binary file not shown.

‎distribution/tools/plugin-cli/build.gradle‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ base {
3737
dependencies {
3838
compileOnly project(":server")
3939
compileOnly project(":libs:opensearch-cli")
40+
api project(":libs:agent-sm:agent-policy")
4041
api "org.bouncycastle:bc-fips:${versions.bouncycastle_jce}"
4142
api "org.bouncycastle:bcpg-fips:${versions.bouncycastle_pg}"
4243
testImplementation project(":test:framework")

‎distribution/tools/plugin-cli/src/main/java/org/opensearch/tools/cli/plugin/PluginSecurity.java‎

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -37,16 +37,15 @@
3737
import org.opensearch.cli.Terminal.Verbosity;
3838
import org.opensearch.cli.UserException;
3939
import org.opensearch.common.util.io.IOUtils;
40+
import org.opensearch.secure_sm.policy.PolicyFile;
4041

4142
import java.io.IOException;
4243
import java.nio.file.Files;
4344
import java.nio.file.Path;
44-
import java.security.NoSuchAlgorithmException;
4545
import java.security.Permission;
4646
import java.security.PermissionCollection;
4747
import java.security.Permissions;
4848
import java.security.Policy;
49-
import java.security.URIParameter;
5049
import java.security.UnresolvedPermission;
5150
import java.util.ArrayList;
5251
import java.util.Collections;
@@ -143,22 +142,12 @@ static Set<String> parsePermissions(Path file, Path tmpDir) throws IOException {
143142
// 2. read permission to the code itself (e.g. jar file of the code)
144143

145144
Path emptyPolicyFile = Files.createTempFile(tmpDir, "empty", "tmp");
146-
final Policy emptyPolicy;
147-
try {
148-
emptyPolicy = Policy.getInstance("JavaPolicy", new URIParameter(emptyPolicyFile.toUri()));
149-
} catch (NoSuchAlgorithmException e) {
150-
throw new RuntimeException(e);
151-
}
145+
final Policy emptyPolicy = new PolicyFile(emptyPolicyFile.toUri().toURL());
152146
IOUtils.rm(emptyPolicyFile);
153147

154148
// parse the plugin's policy file into a set of permissions
155-
final Policy policy;
156-
try {
157-
policy = Policy.getInstance("JavaPolicy", new URIParameter(file.toUri()));
158-
} catch (NoSuchAlgorithmException e) {
159-
throw new RuntimeException(e);
160-
}
161-
PermissionCollection permissions = policy.getPermissions(PluginSecurity.class.getProtectionDomain());
149+
final Policy policy = new PolicyFile(file.toUri().toURL());
150+
final PermissionCollection permissions = policy.getPermissions(PluginSecurity.class.getProtectionDomain());
162151
// this method is supported with the specific implementation we use, but just check for safety.
163152
if (permissions == Policy.UNSUPPORTED_EMPTY_COLLECTION) {
164153
throw new UnsupportedOperationException("JavaPolicy implementation does not support retrieving permissions");

0 commit comments

Comments
 (0)