Skip to content

[ML] Force time shift integration test#111620

Merged
valeriy42 merged 9 commits intoelastic:mainfrom
valeriy42:force-time-shift-it
Aug 9, 2024
Merged

[ML] Force time shift integration test#111620
valeriy42 merged 9 commits intoelastic:mainfrom
valeriy42:force-time-shift-it

Conversation

@valeriy42
Copy link
Copy Markdown
Contributor

This PR adds an integration test to verify that the force_time_shift detection rule works as expected.

…#110582)

DO NOT MERGE THIS INTO `main`!

This PR enables the storage of data and configuration of an anomaly detection job in files so it can be reproduced using the `autodetect` process without Elasticsearch.

To enable the storage, specify `keep_job_data` parameter in the `custom_settings` parameter of the job config:

```json
  "custom_settings": {
    "keep_job_data": "true"
    } 
```

Now, start the job and watch for a log message with the autodetect command similar to the following:

```bash
[2024-06-19T16:03:38,248][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [Elastic-MBP.fritz.box] Autodetect process command: [./autodetect, --lengthEncodedInput, --maxAnomalyRecords=500, --validElasticLicenseKeyConfirmed=true, --config=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/config10764979302390040373.json, --logPipe=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_log_45530, --input=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_input_45530, --inputIsPipe, --output=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_output_45530, --outputIsPipe, --persist=/var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_persist_45530, --persistIsPipe, --namedPipeConnectTimeout=10]
```
and
```bash
[2024-06-19T15:29:08,640][INFO ][o.e.x.m.p.w.LengthEncodedWriter]  Opening file: /var/folders/_j/gcj6z4b950bdzpw7_fzrmpf40000gn/T/elasticsearch-12668972032551307591/autodetect_test-2_input_45530 for writing.
```

Copy the config file, the persist file from the first message, and the input file from the second message.
@valeriy42 valeriy42 requested a review from a team as a code owner August 6, 2024 09:26
@valeriy42 valeriy42 added >test Issues or PRs that are addressing/adding tests v8.16.0 :ml Machine learning labels Aug 6, 2024
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine elasticsearchmachine added the Team:ML Meta label for the ML team label Aug 6, 2024
Copy link
Copy Markdown
Member

@davidkyle davidkyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

valeriy42 and others added 2 commits August 6, 2024 12:54
…ava/org/elasticsearch/xpack/ml/integration/DetectionRulesIT.java

Co-authored-by: David Kyle <david.kyle@elastic.co>
@valeriy42 valeriy42 merged commit d19d250 into elastic:main Aug 9, 2024
@valeriy42 valeriy42 deleted the force-time-shift-it branch August 9, 2024 10:30
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Aug 9, 2024
* upstream/main: (22 commits)
  Prune changelogs after 8.15.0 release
  Bump versions after 8.15.0 release
  EIS integration (elastic#111154)
  Skip LOOKUP/INLINESTATS cases unless on snapshot (elastic#111755)
  Always enforce strict role validation (elastic#111056)
  Mute org.elasticsearch.xpack.esql.analysis.VerifierTests testUnsupportedAndMultiTypedFields elastic#111753
  [ML] Force time shift integration test (elastic#111620)
  ESQL: Add tests for sort, where with unsupported type (elastic#111737)
  [ML] Force time shift documentation (elastic#111668)
  Fix remote cluster credential secure settings reload   (elastic#111535)
  ESQL: Fix for overzealous validation in case of invalid mapped fields (elastic#111475)
  Pass allow security manager flag in gradle test policy setup plugin (elastic#111725)
  Rename streamContent/Separator to bulkContent/Separator (elastic#111716)
  Mute org.elasticsearch.tdigest.ComparisonTests testSparseGaussianDistribution elastic#111721
  Remove 8.14 from branches.json
  Only emit product origin in deprecation log if present (elastic#111683)
  Forward port release notes for v8.15.0 (elastic#111714)
  [ES|QL] Combine Disjunctive CIDRMatch (elastic#111501)
  ESQL: Remove qualifier from attrs (elastic#110581)
  Force using the last centroid during merging (elastic#111644)
  ...

# Conflicts:
#	server/src/main/java/org/elasticsearch/TransportVersions.java
#	x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceNamedWriteablesProvider.java
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this pull request Sep 4, 2024
This PR adds an integration test to verify that the force_time_shift detection rule works as expected.
davidkyle pushed a commit to davidkyle/elasticsearch that referenced this pull request Sep 5, 2024
This PR adds an integration test to verify that the force_time_shift detection rule works as expected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:ml Machine learning Team:ML Meta label for the ML team >test Issues or PRs that are addressing/adding tests v8.16.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants