Skip to content

[5.x] [esArchiver] combine elasticDump and ScenarioManager (#10359)#11170

Closed
spalger wants to merge 2 commits intoelastic:5.xfrom
spalger:backport/5.x/10359
Closed

[5.x] [esArchiver] combine elasticDump and ScenarioManager (#10359)#11170
spalger wants to merge 2 commits intoelastic:5.xfrom
spalger:backport/5.x/10359

Conversation

@spalger
Copy link
Copy Markdown
Contributor

@spalger spalger commented Apr 11, 2017

Backport of #10359

  • As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.

Differences:

  • Streaming implementation allows for much larger archives
  • CLI for creating and using archives
  • Configurable archive location
  • Stores the data in gzipped files (better for source control, searching, large archives)
  • Automatically identifies and upgrades Kibana config documents

Methods:

  • #load(name): import an archive
  • #loadIfNeeded(name): import an archive, but skip the documents what belong to any existing index
  • #unload(name): delete the indexes stored in an archive

CLI operations:

  • ./bin/es_archiver save <name> [index patterns...]: save the mapping and documents in one or more indexes that match the wild-card patterns into an the <name> archive
  • ./bin/es_archiver load <name>: load the mapping and documents from the <name> archive
  • [functional_tests/common/nagivate] check for statusPage

  • [es_archiver] move bins into new scripts dir

  • [functional_tests/apps/context] use esArchiver

  • [esArchiver] general improvements after showing to a few folks

  • remove auto-upgrading config doc logic (until we have better access to kibana version info)
  • export unload command
  • remove preemptive checks in favor of reacting to errors
  • use type "doc" vs "hit" for doc records (consistency)
  • wrote a bunch of pending tests to think though and plan
  • [esArchiver] make log a stream that writes to itself

  • [esArchiver] fill in stats and archive format tests

  • [esArchiver] splitup action logic

  • [esArchiver/cli] fix cli --help output and comment

  • [esArchiver] remove type-based param coercion

  • [esArchiver/log] use strings for log levels

  • [esArchvier] remove unused var

  • [esArchiver/indexDocRecordsStream] add tests

  • [esArchive] fill in remaining tests

  • [esArchiver] fix dem tests

  • [eslint] remove unused vars

  • [esArchiver/loadIfNeeded] fix call to load()

  • [esArchiver] remove loadDumpData helpers

spalger added 2 commits April 11, 2017 15:28
* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.

Differences:
  - Streaming implementation allows for much larger archives
  - CLI for creating and using archives
  - Configurable archive location
  - Stores the data in gzipped files (better for source control, searching, large archives)
  - Automatically identifies and upgrades Kibana config documents

Methods:
  - `#load(name)`: import an archive
  - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index
  - `#unload(name)`: delete the indexes stored in an archive

CLI operations:
  - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive
  - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive

* [functional_tests/common/nagivate] check for statusPage

* [es_archiver] move bins into new scripts dir

* [functional_tests/apps/context] use esArchiver

* [esArchiver] general improvements after showing to a few folks

 - remove auto-upgrading config doc logic (until we have better access to kibana version info)
 - export unload command
 - remove preemptive checks in favor of reacting to errors
 - use type "doc" vs "hit" for doc records (consistency)
 - wrote a bunch of pending tests to think though and plan

* [esArchiver] make log a stream that writes to itself

* [esArchiver] fill in stats and archive format tests

* [esArchiver] splitup action logic

* [esArchiver/cli] fix cli --help output and comment

* [esArchiver] remove type-based param coercion

* [esArchiver/log] use strings for log levels

* [esArchvier] remove unused var

* [esArchiver/indexDocRecordsStream] add tests

* [esArchive] fill in remaining tests

* [esArchiver] fix dem tests

* [eslint] remove unused vars

* [esArchiver/loadIfNeeded] fix call to load()

* [esArchiver] remove loadDumpData helpers
@spalger spalger added backport This PR is a backport of another PR v5.4.0 labels Apr 11, 2017
@spalger
Copy link
Copy Markdown
Contributor Author

spalger commented Apr 11, 2017

Superseded by #11171

@spalger spalger closed this Apr 11, 2017
@spalger spalger deleted the backport/5.x/10359 branch August 18, 2020 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR v5.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant