[esArchiver] combine elasticDump and ScenarioManager by spalger · Pull Request #10359 · elastic/kibana

spalger · 2017-02-15T03:00:04Z

Builds on top of #10187
A dependency of #9853

As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.

Differences:

Streaming implementation allows for much larger archives
CLI for creating and using archives
Configurable archive location
Stores the data in gzipped files (better for source control, searching, large archives)
Automatically identifies and upgrades Kibana config documents

Methods:

#load(name): import an archive
#loadIfNeeded(name): import an archive, but skip the documents what belong to any existing index
#unload(name): delete the indexes stored in an archive
#save(name, indices): programmatically save indices to an archive (using the CLI is preferred)

CLI operations:

node scripts/es_archiver save <name> [index patterns...]
- save the mapping and documents in one or more indexes that match the wild-card patterns into an the <name> archive
node scripts/es_archiver load <name>
- load the mapping and documents from the <name> archive
node scripts/es_archiver --help
- see other commands/options

… these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive

…archiver

kimjoar · 2017-02-22T04:33:18Z

To review this I think we need to zoom. I'm mostly busy tomorrow, but maybe the day after? Just sync up so you can show me the important stuff, then I can take a deeper look.

…archiver

spalger · 2017-02-23T18:37:27Z

jenkins, test this

- remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan

spalger · 2017-02-24T01:13:53Z

@epixa @kjbekkelund Tests are passing and I'm working on writing more tests for the added functionality but it's ready to look at I think

…archiver

spalger · 2017-03-22T02:01:25Z

@epixa @kjbekkelund pushed updates to address feedback if you don't mind taking a look.

epixa · 2017-03-22T11:54:19Z

The tests for esArchiver are failing on jenkins

…archiver

kimjoar · 2017-03-22T19:03:17Z

I don't immediately see any big problems with the changes, so LGTM. (CI is struggling, though)

spalger · 2017-03-23T08:01:33Z

@epixa the CI gods have blessed this build, hurry, look now!

epixa · 2017-03-25T15:20:37Z

Before merging, please remove both load_dump_data files (the script wrapper and the actual file), and please update the PR description to accurately reflect the changes to the implementation from the PR review.

…archiver

epixa · 2017-03-27T19:29:11Z

I'm going to merge this because it'll unblock CI builds that are currently failing due to elasticdump (which this removes entirely).

* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers

…#10890) * Removing derivative behavior... adding better tooltips to add delete * Re-organizing metrics; Adding headers; Adding default behavoir * Adding help text * Additional fixes - Fix for deactivated still being able to be added - Added Experimental Feature to description * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true (#10855) * [elasticsearch/healthCheck] ensure that multi.allow_explicit_index=true * [elasticsearch/healthCheck] fix tests * [eslint/console] enable no-undef rule (#10881) * [eslint/console] enable no-undef rule * [eslint/console] fix no-undef violations * Navbarextensions improvements (#9871) * - added controller option - added $location service to determine on which the user is at currently * added test for the hideButton method that is dependent on the location path * removed argument * use locals object to pass variables to the template's scope * use map object * fixing percentage mode extents (#10843) * fixes axis title for new axis (#10866) * fixes metrics options matching (#10865) * Add kuiLocalTab-isDisabled state. (#10830) * Add kuiLocalTab-isDisabled state. * Update Management header to use kuiLocalTab-isDisabled class. * test: send content-type with proxy POST tests (#10903) Elasticsearch now requires that content-type be sent on all requests with payloads, so our tests should be sending it with requests as well. * [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers

* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers

@gammon

* [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers * [esArchiver] update archives for 5.x * [functionalTestRunner] replace intern (#10910) * [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task * [eslint] remove use of context()

spalger force-pushed the implement/es-archiver branch from e429ff1 to 2017ff8 Compare February 15, 2017 03:01

spalger mentioned this pull request Feb 15, 2017

WIP: [functional-tests] replace intern with custom test runner #9853

Closed

spalger force-pushed the implement/es-archiver branch 2 times, most recently from 197e4cd to 09fe2c8 Compare February 15, 2017 03:49

spalger added Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// review v5.4.0 v6.0.0 labels Feb 15, 2017

spalger requested review from epixa and kimjoar February 15, 2017 03:54

spalger force-pushed the implement/es-archiver branch from 09fe2c8 to 0d869b9 Compare February 15, 2017 04:32

spalger force-pushed the implement/es-archiver branch from 0d869b9 to 70fc947 Compare February 15, 2017 04:59

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

28c34f2

…archiver

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

3b6676c

…archiver

spalger force-pushed the implement/es-archiver branch from 52ecc50 to 3b6676c Compare February 23, 2017 00:47

[functional_tests/common/nagivate] check for statusPage

923636b

spalger added 3 commits February 23, 2017 13:04

[es_archiver] move bins into new scripts dir

86af5a4

[functional_tests/apps/context] use esArchiver

8c77b43

spalger force-pushed the implement/es-archiver branch from 7561500 to ca6ddc3 Compare February 24, 2017 03:15

[esArchiver] make log a stream that writes to itself

30bdefc

spalger force-pushed the implement/es-archiver branch from ca6ddc3 to 79add82 Compare February 24, 2017 04:56

[esArchiver] fill in stats and archive format tests

edfe4b3

spalger force-pushed the implement/es-archiver branch from 79add82 to edfe4b3 Compare February 24, 2017 05:01

spalger added 2 commits February 24, 2017 01:21

[esArchiver] splitup action logic

01c0dcb

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

5e602e8

…archiver

spalger added 3 commits March 21, 2017 15:51

[esArchvier] remove unused var

740a466

[esArchiver/indexDocRecordsStream] add tests

d50983f

[esArchive] fill in remaining tests

f039afd

spalger added 3 commits March 22, 2017 07:04

[esArchiver] fix dem tests

f0e0130

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

d8f85f5

…archiver

[eslint] remove unused vars

1ec2a05

[esArchiver/loadIfNeeded] fix call to load()

b926d96

epixa approved these changes Mar 25, 2017

View reviewed changes

spalger added 3 commits March 27, 2017 09:26

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

eefb8cc

…archiver

[esArchiver] remove loadDumpData helpers

9e2ee74

Merge branch 'master' of github.com:elastic/kibana into implement/es-…

9449535

…archiver

epixa merged commit e677900 into elastic:master Mar 27, 2017

spalger mentioned this pull request Mar 28, 2017

[functionalTestRunner] replace intern #10910

Merged

2 tasks

spalger deleted the implement/es-archiver branch March 29, 2017 19:14

spalger mentioned this pull request Apr 11, 2017

[5.x] [esArchiver] combine elasticDump and ScenarioManager (#10359) #11170

Closed

jsanchez91 mentioned this pull request Jul 2, 2019

Add E2E and unit tests to the front-end side and improve back-end tests wazuh/wazuh-dashboard-plugins#1539

Open

12 tasks

sorenlouv mentioned this pull request Aug 25, 2020

[APM] Use the outcome field to calculate the transaction error rate chart #75528

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[esArchiver] combine elasticDump and ScenarioManager#10359

[esArchiver] combine elasticDump and ScenarioManager#10359
epixa merged 25 commits intoelastic:masterfrom
spalger:implement/es-archiver

spalger commented Feb 15, 2017 •

edited

Loading

Uh oh!

kimjoar commented Feb 22, 2017

Uh oh!

spalger commented Feb 23, 2017

Uh oh!

spalger commented Feb 24, 2017

Uh oh!

spalger commented Mar 22, 2017

Uh oh!

epixa commented Mar 22, 2017

Uh oh!

kimjoar commented Mar 22, 2017

Uh oh!

spalger commented Mar 23, 2017

Uh oh!

epixa commented Mar 25, 2017

Uh oh!

epixa commented Mar 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

spalger commented Feb 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kimjoar commented Feb 22, 2017

Uh oh!

spalger commented Feb 23, 2017

Uh oh!

spalger commented Feb 24, 2017

Uh oh!

spalger commented Mar 22, 2017

Uh oh!

epixa commented Mar 22, 2017

Uh oh!

kimjoar commented Mar 22, 2017

Uh oh!

spalger commented Mar 23, 2017

Uh oh!

epixa commented Mar 25, 2017

Uh oh!

epixa commented Mar 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

spalger commented Feb 15, 2017 •

edited

Loading