[Maps] get max_result_window and max_inner_result_window from index settings by nreese · Pull Request #53500 · elastic/kibana

nreese · 2019-12-18T15:54:06Z

This PR updates _search requests to use max_result_window value from index rather then hard coded 10000 limit.

max_inner_result_window used to allow users to configure larger number of top hits.

To test, increase, or decrease, max_result_window with the following command.

PUT /kibana_sample_data_logs/_settings
{
  "index.max_result_window" : "15000"
}

…ndow

elasticmachine · 2019-12-18T15:54:08Z

Pinging @elastic/kibana-gis (Team:Geo)

nreese · 2019-12-18T22:19:21Z

One problem with pulling max_result_window from indexSettings is that users can have read access to the index without having read access to index settings. Index settings read access requires view_index_metadata role. In these cases, users will be defaulted to static value like the current behavior.

elasticmachine · 2019-12-19T01:43:41Z

Pinging @elastic/kibana-docs (Team:Docs)

docs/maps/maps-aggregations.asciidoc

docs/maps/vector-layer.asciidoc

Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com>

jsanz · 2019-12-23T11:06:33Z

Tested both (chromium and firefox) to push the limit of 10K but also to restrict and works nicely on both cases. Great to see how far the Maps application can go.

nreese · 2019-12-26T13:58:44Z

@elasticmachine merge upstream

kibanamachine · 2019-12-26T15:19:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c1e6ced

History

💚 Build #16806 succeeded 384d383
💚 Build #16805 succeeded aa9a6ac
💚 Build #16597 succeeded 369c66b
💔 Build #16576 failed 3a7f27e
💔 Build #16481 failed eb168a5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2019-12-26T17:29:13Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 1072d23

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

thomasneirynck

some minor comments.

x-pack/legacy/plugins/maps/public/layers/sources/es_search_source/load_index_settings.js

x-pack/legacy/plugins/maps/server/routes.js

x-pack/legacy/plugins/maps/public/layers/sources/es_search_source/es_search_source.js

nreese · 2020-01-02T13:19:04Z

@elasticmachine merge upstream

kibanamachine · 2020-01-02T16:18:18Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1bafa59

History

💚 Build #17559 succeeded ac1bc63
💚 Build #17278 succeeded 2770545

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

thomasneirynck

thx. this is a nice pathway for making the 10k limit configurable 🎩 . It is an approach which doesn't clash with underlying es-cluster settings.

thomasneirynck · 2020-01-02T17:06:41Z

x-pack/legacy/plugins/maps/public/layers/sources/es_search_source/load_index_settings.js

+const indexSettings = new Map();
+
 export async function loadIndexSettings(indexPatternTitle) {
+  if (indexSettings.has(indexPatternTitle)) {


good call caching the promise, not the result.

…ettings (elastic#53500) * [Maps] pull ES_SIZE_LIMIT and top hits limit from index settings * get fetch working * get min values from indicies response * use indexSettings.maxResultWindow in documents request size * use max_inner_result_window to define top hits max * update jest test * update docs * more docs changes for top hits * fix line spacing * Update docs/maps/maps-aggregations.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/maps/vector-layer.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * add api integration test for indexSettings route * eslint fixes * review feedback * display toast on first index settings fetch failure * clean up Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…ettings (#53500) (#53902) * [Maps] pull ES_SIZE_LIMIT and top hits limit from index settings * get fetch working * get min values from indicies response * use indexSettings.maxResultWindow in documents request size * use max_inner_result_window to define top hits max * update jest test * update docs * more docs changes for top hits * fix line spacing * Update docs/maps/maps-aggregations.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/maps/vector-layer.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * add api integration test for indexSettings route * eslint fixes * review feedback * display toast on first index settings fetch failure * clean up Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use handlebars@4.5.3 (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877)

* master: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use handlebars@4.5.3 (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

…ris/kibana into alerting/created_at-and-updated_at * 'alerting/created_at-and-updated_at' of github.com:gmmorris/kibana: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use handlebars@4.5.3 (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

…t-types * alerting/created_at-and-updated_at: updatedAt should equal createdAt on creation Move index patterns: src/legacy/core_plugins/data 👉 src/plugins/data (elastic#53794) moved Task Manager server code under "server" directory (elastic#53777) Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use handlebars@4.5.3 (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

nreese added 3 commits December 17, 2019 12:32

[Maps] pull ES_SIZE_LIMIT and top hits limit from index settings

45418fc

Merge branch 'master' of github.com:elastic/kibana into max_result_wi…

43f05fc

…ndow

get fetch working

14e1865

nreese added WIP Work in progress release_note:fix Team:Geo Former Team Label for Geo Team. Now use Team:Presentation v8.0.0 v7.6.0 labels Dec 18, 2019

nreese added 2 commits December 18, 2019 08:59

get min values from indicies response

eb168a5

use indexSettings.maxResultWindow in documents request size

2c7fb06

nreese removed the WIP Work in progress label Dec 18, 2019

nreese requested a review from thomasneirynck December 18, 2019 22:18

use max_inner_result_window to define top hits max

3a7f27e

nreese requested a review from kindsun December 18, 2019 22:40

nreese added 3 commits December 18, 2019 18:35

update jest test

a4f9d22

update docs

3fd34c3

more docs changes for top hits

4628ca2

nreese added the Team:Docs label Dec 19, 2019

nreese added release_note:enhancement and removed release_note:fix labels Dec 19, 2019

nreese requested a review from gchaps December 19, 2019 01:43

fix line spacing

369c66b

gchaps reviewed Dec 19, 2019

View reviewed changes

docs/maps/maps-aggregations.asciidoc Outdated Show resolved Hide resolved

docs/maps/vector-layer.asciidoc Outdated Show resolved Hide resolved

Update docs/maps/maps-aggregations.asciidoc

aa9a6ac

Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com>

nreese requested a review from a team as a code owner December 19, 2019 22:01

Update docs/maps/vector-layer.asciidoc

384d383

Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com>

Merge branch 'master' into max_result_window

c1e6ced

elastic deleted a comment from elasticmachine Dec 26, 2019

add api integration test for indexSettings route

1072d23

eslint fixes

2770545

thomasneirynck reviewed Dec 30, 2019

View reviewed changes

elasticmachine and others added 4 commits January 2, 2020 05:19

Merge branch 'master' into max_result_window

ac1bc63

review feedback

58f8051

display toast on first index settings fetch failure

3a6a8fa

clean up

1bafa59

nreese requested review from gchaps and thomasneirynck January 2, 2020 15:02

thomasneirynck approved these changes Jan 2, 2020

View reviewed changes

nreese merged commit 47e5342 into elastic:master Jan 2, 2020

nreese mentioned this pull request Jan 2, 2020

[7.x] [Maps] get max_result_window and max_inner_result_window from index settings (#53500) #53902

Merged

nreese mentioned this pull request Jan 5, 2020

More than 10,000 documents in map plot #53959

Closed

spong mentioned this pull request Jan 7, 2020

[Maps] [SIEM] Map shows error requesting 'view_index_metadata' permissions but functions normally #54085

Closed

thomasneirynck mentioned this pull request Jan 22, 2020

[Maps] Consider options beyond aggregations for obtaining es search results docs over the 10k limit #49971

Closed

nreese mentioned this pull request Jan 26, 2020

[Lens] Provide a datasource for individual documents #55897

Closed

Conversation

nreese commented Dec 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Dec 18, 2019

Uh oh!

nreese commented Dec 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Dec 19, 2019

Uh oh!

Uh oh!

Uh oh!

jsanz commented Dec 23, 2019

Uh oh!

nreese commented Dec 26, 2019

Uh oh!

kibanamachine commented Dec 26, 2019

💚 Build Succeeded

History

Uh oh!

kibanamachine commented Dec 26, 2019

💔 Build Failed

Uh oh!

thomasneirynck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nreese commented Jan 2, 2020

Uh oh!

kibanamachine commented Jan 2, 2020

💚 Build Succeeded

History

Uh oh!

thomasneirynck left a comment

Choose a reason for hiding this comment

Uh oh!

thomasneirynck Jan 2, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nreese commented Dec 18, 2019 •

edited

Loading

nreese commented Dec 18, 2019 •

edited

Loading