Skip to content

[7.x] [File upload] Migrate routing to NP & add route validation (#52313)#57063

Merged
kindsun merged 1 commit intoelastic:7.xfrom
kindsun:backport/7.x/pr-52313
Feb 7, 2020
Merged

[7.x] [File upload] Migrate routing to NP & add route validation (#52313)#57063
kindsun merged 1 commit intoelastic:7.xfrom
kindsun:backport/7.x/pr-52313

Conversation

@kindsun
Copy link
Copy Markdown
Contributor

@kindsun kindsun commented Feb 6, 2020

Backports the following commits to 7.x:

[File upload] Migrate routing to NP & add route validation (elastic#5…
17ca4ba 
…2313)

* Cursory validation

* Handle empty data arrays and settings conditionally

* Set validation defaults. Move logic to routes folder and separate for testing

* Move plugin init back into routes folder. Syntax updates

* Migrate router to NP

* Use new np router and fetch. Add placeholder schema validation

* Override default 'maxBytes'

* Body with first-level schema keys in place

* Add conditional validation of mappings, data and settings. Clean up old joi code

* Ensure query is handled correctly on both sides. Iron out decision logic on server-side

* Move conditional validation to first step in payload handling

* Update http_service to work with latest NP changes on master

* Some reorg. Only update telemetry if no errors

* Clean up

* Test file upload validation logic

* Linting

* Review feedback. Remove unneeded apiBasePath var

* Pass entire req object with through to ES, not just the validated fields
@kindsun kindsun added the backport This PR is a backport of another PR label Feb 6, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Feb 7, 2020

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/rollup_job/tsvb·js.rollup job tsvb integration create rollup tsvb

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://github.com/elastic/kibana/issues/57065

[00:00:00]       │
[00:29:30]         └-: rollup job
[00:29:30]           └-> "before all" hook
[00:30:52]           └-: tsvb integration
[00:30:52]             └-> "before all" hook
[00:30:52]             └-> "before all" hook
[00:30:52]               │ info [visualize/default] Loading "mappings.json"
[00:30:52]               │ info [visualize/default] Loading "data.json"
[00:30:52]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/Fq1FWOX8SFmUSPTHBlXgxQ] deleting index
[00:30:52]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_1/7uarmfIkSTOY4PJQqff87w] deleting index
[00:30:52]               │ info [visualize/default] Deleted existing index [".kibana_2",".kibana_1"]
[00:30:52]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [doc]
[00:30:53]               │ info [visualize/default] Created index ".kibana"
[00:30:53]               │ debg [visualize/default] ".kibana" settings {"index":{"number_of_shards":"1","auto_expand_replicas":"0-1","number_of_replicas":"0"}}
[00:30:53]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [metricbeat-7] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:30:53]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [metricbeat-7/iE6VMNYXQxOocmxQKhI8Dw] create_mapping [_doc]
[00:30:53]               │ info [visualize/default] Indexed 8 docs into ".kibana"
[00:30:53]               │ info [visualize/default] Indexed 1 docs into "metricbeat-7"
[00:30:54]               │ info Creating index .kibana_2.
[00:30:54]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:30:54]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] updating number_of_replicas to [0] for indices [.kibana_2]
[00:30:54]               │ info Reindexing .kibana to .kibana_1
[00:30:54]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:30:54]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] updating number_of_replicas to [0] for indices [.kibana_1]
[00:30:54]               │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] 70822 finished with response BulkByScrollResponse[took=89.2ms,timed_out=false,sliceId=null,updated=0,created=8,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:30:54]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana/0WrAOUSdSDe2eCEMk2Cg2w] deleting index
[00:30:54]               │ info Migrating .kibana_1 saved objects to .kibana_2
[00:30:54]               │ debg Migrating saved objects space:default, index-pattern:metricbeat-*, custom-space:index-pattern:metricbeat-*, index-pattern:logstash-*, custom_space:index-pattern:logstash-*, visualization:i-exist, custom_space:visualization:i-exist, query:okjpgs
[00:30:54]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/3vM2gPOvQ0K5jdpAafhCoA] update_mapping [_doc]
[00:30:54]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/3vM2gPOvQ0K5jdpAafhCoA] update_mapping [_doc]
[00:30:54]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/3vM2gPOvQ0K5jdpAafhCoA] update_mapping [_doc]
[00:30:54]               │ info Pointing alias .kibana to .kibana_2.
[00:30:54]               │ info Finished in 850ms.
[00:30:54]               │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:30:55]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/3vM2gPOvQ0K5jdpAafhCoA] update_mapping [_doc]
[00:30:56]             └-> create rollup tsvb
[00:30:56]               └-> "before each" hook: global before each
[00:30:56]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [rollup-source-data] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:30:56]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [rollup-source-data/2zQafg0bQwKjFp_tFfXSgQ] create_mapping [_doc]
[00:30:56]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [rollup-target-data] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:30:56]               │ debg navigating to visualize url: http://localhost:6121/app/kibana#/visualize
[00:30:56]               │ debg Navigate to: http://localhost:6121/app/kibana#/visualize
[00:30:56]               │ info [o.e.x.r.j.RollupJobTask] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] Rollup job [tsvb-test-rollup-job-1581033522265] created.
[00:30:56]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [rollup-target-data/miJLzQtRQxejxQ2SySdRVA] update_mapping [_doc]
[00:30:56]               │ debg ... sleep(700) start
[00:30:56]               │ debg browser[INFO] http://localhost:6121/app/kibana?_t=1581035381074#/visualize 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:30:56]               │
[00:30:56]               │ debg browser[INFO] http://localhost:6121/bundles/app/kibana/bootstrap.js 8:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:30:57]               │ debg ... sleep(700) end
[00:30:57]               │ debg returned from get, calling refresh
[00:30:57]               │ debg browser[INFO] http://localhost:6121/app/kibana?_t=1581035381074#/visualize 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:30:57]               │
[00:30:57]               │ debg browser[INFO] http://localhost:6121/bundles/app/kibana/bootstrap.js 8:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:30:57]               │ debg currentUrl = http://localhost:6121/app/kibana#/visualize
[00:30:57]               │          appUrl = http://localhost:6121/app/kibana#/visualize
[00:30:57]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:31:02]               │ debg TestSubjects.find(kibanaChrome)
[00:31:02]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=10000
[00:31:02]               │ debg browser[INFO] http://localhost:6121/built_assets/dlls/vendors_3.bundle.dll.js 520:138197 "INFO: 2020-02-07T00:29:45Z
[00:31:02]               │        Adding connection to http://localhost:6121/elasticsearch
[00:31:02]               │
[00:31:02]               │      "
[00:31:02]               │ debg ... sleep(501) start
[00:31:02]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581031728131548899] [.kibana_2/3vM2gPOvQ0K5jdpAafhCoA] update_mapping [_doc]
[00:31:02]               │ debg ... sleep(501) end
[00:31:02]               │ debg in navigateTo url = http://localhost:6121/app/kibana#/visualize?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:31:02]               │ debg TestSubjects.exists(statusPageContainer)
[00:31:02]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:31:05]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:31:05]               │ debg TestSubjects.exists(newItemButton)
[00:31:05]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="newItemButton"]') with timeout=2500
[00:31:05]               │ debg TestSubjects.click(newItemButton)
[00:31:05]               │ debg Find.clickByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:31:05]               │ debg Find.findByCssSelector('[data-test-subj="newItemButton"]') with timeout=10000
[00:31:06]               │ debg TestSubjects.find(visNewDialogTypes)
[00:31:06]               │ debg Find.findByCssSelector('[data-test-subj="visNewDialogTypes"]') with timeout=10000
[00:31:06]               │ debg TestSubjects.click(visType-metrics)
[00:31:06]               │ debg Find.clickByCssSelector('[data-test-subj="visType-metrics"]') with timeout=10000
[00:31:06]               │ debg Find.findByCssSelector('[data-test-subj="visType-metrics"]') with timeout=10000
[00:31:06]               │ debg isGlobalLoadingIndicatorVisible
[00:31:06]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:31:06]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:31:08]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:31:08]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:31:08]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:31:08]               │ debg TestSubjects.exists(tvbVisEditor)
[00:31:08]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="tvbVisEditor"]') with timeout=10000
[00:31:08]               │ debg openQuickSelectTimeMenu
[00:31:08]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:31:08]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=2500
[00:31:11]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:31:11]               │ debg opening quick select menu
[00:31:11]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:31:11]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:31:11]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:31:11]               │ debg TestSubjects.click(superDatePickerCommonlyUsed_Last_24 hours)
[00:31:11]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerCommonlyUsed_Last_24 hours"]') with timeout=10000
[00:31:11]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerCommonlyUsed_Last_24 hours"]') with timeout=10000
[00:31:12]               │ debg TestSubjects.find(metricTsvbTypeBtn)
[00:31:12]               │ debg Find.findByCssSelector('[data-test-subj="metricTsvbTypeBtn"]') with timeout=10000
[00:31:12]               │ debg TestSubjects.exists(tsvbMetricValue)
[00:31:12]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="tsvbMetricValue"]') with timeout=10000
[00:31:13]               │ debg TestSubjects.click(metricEditorPanelOptionsBtn)
[00:31:13]               │ debg Find.clickByCssSelector('[data-test-subj="metricEditorPanelOptionsBtn"]') with timeout=10000
[00:31:13]               │ debg Find.findByCssSelector('[data-test-subj="metricEditorPanelOptionsBtn"]') with timeout=10000
[00:31:13]               │ debg isGlobalLoadingIndicatorVisible
[00:31:13]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:31:13]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:31:14]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:31:15]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:31:15]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:31:15]               │ debg TestSubjects.find(metricsIndexPatternInput)
[00:31:15]               │ debg Find.findByCssSelector('[data-test-subj="metricsIndexPatternInput"]') with timeout=10000
[00:31:15]               │ debg isGlobalLoadingIndicatorVisible
[00:31:15]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:31:15]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:31:17]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:31:17]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:31:17]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:31:17]               │ debg TestSubjects.find(metricsIndexPatternInterval)
[00:31:17]               │ debg Find.findByCssSelector('[data-test-subj="metricsIndexPatternInterval"]') with timeout=10000
[00:31:17]               │ debg isGlobalLoadingIndicatorVisible
[00:31:17]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:31:17]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:31:19]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:31:19]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:31:19]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:31:19]               │ debg TestSubjects.find(metricsDropLastBucket-no)
[00:31:19]               │ debg Find.findByCssSelector('[data-test-subj="metricsDropLastBucket-no"]') with timeout=10000
[00:31:19]               │ debg isGlobalLoadingIndicatorVisible
[00:31:19]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:31:19]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:31:21]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:31:22]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:31:22]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:31:22]               │ debg ... sleep(3000) start
[00:31:25]               │ debg ... sleep(3000) end
[00:31:25]               │ debg Waiting up to 20000ms for rendering count to stabilize...
[00:31:25]               │ debg TestSubjects.find(visualizationLoader)
[00:31:25]               │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
[00:31:25]               │ debg -- firstCount=7
[00:31:25]               │ debg ... sleep(1000) start
[00:31:26]               │ debg ... sleep(1000) end
[00:31:26]               │ debg TestSubjects.find(visualizationLoader)
[00:31:26]               │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
[00:31:26]               │ debg -- secondCount=7
[00:31:26]               │ debg Find.findByCssSelector('.tvbVisMetric__value--primary') with timeout=10000
[00:31:26]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/rollup job tsvb integration create rollup tsvb.png"
[00:31:26]               │ info Current URL is: http://localhost:6121/app/kibana#/visualize/create?type=metrics&_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:%27%27),uiState:(),vis:(aggs:!(),params:(axis_formatter:number,axis_position:left,axis_scale:normal,background_color_rules:!((id:f7a59520-4940-11ea-8741-cbd7b6d0d7e6)),default_index_pattern:%27metricbeat-*%27,default_timefield:%27@timestamp%27,drop_last_bucket:0,id:%2761ca57f0-469d-11e7-af02-69e470af7417%27,index_pattern:rollup-target-data,interval:%271d%27,isModelInvalid:!f,series:!((axis_position:right,chart_type:line,color:%2368BC00,fill:0.5,formatter:number,id:%2761ca57f1-469d-11e7-af02-69e470af7417%27,line_width:1,metrics:!((id:%2761ca57f2-469d-11e7-af02-69e470af7417%27,type:count)),point_size:1,separate_axis:0,split_mode:everything,stacked:none)),show_grid:1,show_legend:1,time_field:%27%27,type:metric),title:%27%27,type:metrics))
[00:31:26]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/rollup job tsvb integration create rollup tsvb.html
[00:31:26]               └- ✖ fail: "rollup job tsvb integration create rollup tsvb"
[00:31:26]               │

Stack Trace

{ Error: expected '0' to sort of equal '3'
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/functional/apps/rollup_job/tsvb.js:90:27)
    at process._tickCallback (internal/process/next_tick.js:68:7) actual: '0', expected: '3', showDiff: true }

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@kindsun kindsun merged commit 0a508b0 into elastic:7.x Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a backport of another PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kindsun @elasticmachine