Skip to content

Commit 8909f2d

Browse files
Merge branch '7.11' into backport/7.11/pr-86093
2 parents 7d9f07b + 15cef72 commit 8909f2d

106 files changed

Lines changed: 3668 additions & 834 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

data/.empty

Whitespace-only changes.

docs/api/upgrade-assistant.asciidoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,6 @@ include::upgrade-assistant/status.asciidoc[]
2222
include::upgrade-assistant/reindexing.asciidoc[]
2323
include::upgrade-assistant/batch_reindexing.asciidoc[]
2424
include::upgrade-assistant/batch_queue.asciidoc[]
25+
include::upgrade-assistant/default-field.asciidoc[]
2526
include::upgrade-assistant/check_reindex_status.asciidoc[]
2627
include::upgrade-assistant/cancel_reindex.asciidoc[]

docs/api/upgrade-assistant/check_reindex_status.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66

77
experimental[] Check the status of the reindex operation.
88

9-
Check the status of the reindex operation.
10-
119
[[check-reindex-status-request]]
1210
==== Request
1311

docs/api/upgrade-assistant/default-field.asciidoc

Lines changed: 31 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,19 @@
11
[[upgrade-assistant-api-default-field]]
2-
=== Add Default Field API
2+
=== Add default field API
3+
++++
4+
<titleabbrev>Add default field</titleabbrev>
5+
++++
36

4-
experimental[This API is *experimental* and may be changed or removed completely in a future release. The underlying Upgrade Assistant concepts are stable, but the APIs for managing Upgrade Assistant are currently experimental.]
7+
experimental[] In {es} 7.0 and later, some query types, such as Simple Query String, have a limit to the number of fields they can query against.
8+
To configure the cap in {es}, set the `indices.query.bool.max_clause_count` cluster setting, which is 1024 by default.
59

6-
Starting in Elasticsearch 7.0, some query types such as Simple Query String, have a limit to the number of fields they will query against. You can configure this cap in Elasticsearch by setting the `indices.query.bool.max_clause_count` cluster setting, which is 1024 by default.
7-
8-
For indices with more fields than this cap, you can add the `index.query.default_field` index setting to inform Elasticsearch which fields to use by default when no field is specified for a query. This API assists in adding this setting to an index in Elasticsearch.
10+
For indices with more fields than the cap, add the `index.query.default_field` index setting to inform {es} which
11+
fields to use by default when no field is specified for a query. Use the add default field API to add the `index.query.default_field` setting to an {es} index.
912

13+
[[upgrade-assistant-api-default-field-request]]
1014
==== Request
1115

12-
To add the `index.query.default_field` setting to an index, submit a POST request to the `/api/upgrade_assistant/add_query_default_field/<index>` endpoint:
13-
14-
Note: You cannot access this endpoint via the Console in Kibana.
16+
To add the `index.query.default_field` setting to an {es} index, submit a POST request to `/api/upgrade_assistant/add_query_default_field/<index>`:
1517

1618
[source,js]
1719
--------------------------------------------------
@@ -23,15 +25,25 @@ GET /api/upgrade_assistant/add_query_default_field/myIndex
2325
--------------------------------------------------
2426
// KIBANA
2527

26-
<1> An array of Elasticsearch field types to use to generate the list of fields. Required.
27-
<2> An array of additional field names, dot-deliminated. Optional.
28+
<1> A required array of {es} field types that generate the list of fields.
29+
<2> An optional array of additional field names, dot-deliminated.
30+
31+
To add the `index.query.default_field` index setting to the specified index, {kib} generates an array of all fields from the index mapping.
32+
The fields contain the types specified in `fieldTypes`. {kib} appends any other fields specified in `otherFields` to the array of default fields.
2833

29-
Kibana will add the `index.query.default_field` index setting to the specified index by generating an array of all fields from the index's mapping that are any of the types specified in `fieldTypes`. Optionally, any other fields specified in `otherFields` will be appended to the array of default fields.
34+
[[upgrade-assistant-api-default-field-response-codes]]
35+
==== Response codes
3036

31-
==== Response
37+
`200`::
38+
Indicates a successful call.
3239

33-
A successful call returns a response code of `200` and a response body
34-
containing a JSON structure similar to the following example:
40+
`400`::
41+
Indicates that the index already has the `index.query.default_field` setting. No changes are made to the index.
42+
43+
[[upgrade-assistant-api-default-field-response-body]]
44+
==== Response body
45+
46+
The response body contains a JSON structure, similar to the following:
3547

3648
[source,js]
3749
--------------------------------------------------
@@ -40,11 +52,10 @@ containing a JSON structure similar to the following example:
4052
}
4153
--------------------------------------------------
4254

43-
If the index already has the `index.query.default_field` setting, Kibana will respond with a 400 Bad Request error and make no changes to the index.
44-
55+
[[upgrade-assistant-api-default-field-example]]
4556
==== Example
4657

47-
For an index with the following mappings:
58+
Your index contains following mappings:
4859

4960
[source,js]
5061
--------------------------------------------------
@@ -68,7 +79,7 @@ GET /myIndex/_mappings
6879
--------------------------------------------------
6980
// CONSOLE
7081

71-
Making this request to Kibana:
82+
Make the following request to {kib}:
7283

7384
[source,js]
7485
--------------------------------------------------
@@ -80,7 +91,7 @@ GET /api/upgrade_assistant/add_query_default_field/myIndex
8091
--------------------------------------------------
8192
// KIBANA
8293

83-
Would result in the `index.query.default_field` setting being added with this value:
94+
The API returns the following:
8495

8596
[source,js]
8697
--------------------------------------------------
@@ -99,4 +110,4 @@ GET /myIndex/_settings?flat_settings=true
99110
--------------------------------------------------
100111
// CONSOLE
101112

102-
Kibana generated the `field1` and `nestedfield.field4` values based on the specified `fieldTypes` and then appended the `otherFields` to the array.
113+
{kib} generates the `field1` and `nestedfield.field4` values based on the specified `fieldTypes`, then appends the `otherFields` to the array.

docs/fleet/fleet.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,4 +24,4 @@ image::fleet/images/fleet-start.png[{fleet} app in {kib}]
2424
== Get started
2525

2626
To get started with {fleet}, refer to the
27-
{ingest-guide}/index.html[{fleet}] docs.
27+
{fleet-guide}/index.html[{fleet}] docs.

docs/getting-started/quick-start-guide.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,4 +140,4 @@ For more information, refer to <<lens, *Lens*>>.
140140

141141
If you are you ready to add your own data, refer to <<connect-to-elasticsearch,Add data to {kib}>>.
142142

143-
If you want to ingest your data, refer to {ingest-guide}/fleet-quick-start.html[Quick start: Get logs and metrics into the Elastic Stack].
143+
If you want to ingest your data, refer to {fleet-guide}/fleet-quick-start.html[Quick start: Get logs and metrics into the Elastic Stack].

docs/maps/connect-to-ems.asciidoc

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,188 @@ To disable EMS, change your <<settings, kibana.yml>> file.
3535
. Set `map.includeElasticMapsService` to `false` to turn off the EMS connection.
3636
. Set `map.tilemap.url` to the URL of your tile server. This configures the default tile layer of Maps.
3737
. (Optional) Set `map.regionmap` to the vector shapes of the administrative boundaries that you want to use.
38+
39+
[float]
40+
[id=elastic-maps-server]
41+
=== Host Elastic Maps Service locally
42+
43+
beta::[]
44+
45+
If you cannot connect to Elastic Maps Service from the {kib} server or browser clients, and your cluster has the appropriate license level, you can opt to host the service on your own infrastructure.
46+
47+
{hosted-ems} is a self-managed version of Elastic Maps Service offered as a Docker image that provides both the EMS basemaps and EMS boundaries. You must first download and run the image. After connecting it to your {es} cluster for license validation, you're guided to download and configure the basemaps database, which must be retrieved separately.
48+
49+
IMPORTANT: {hosted-ems} does not serve raster tiles, needed by Vega, coordinate, and region map visualizations.
50+
51+
You can use +docker pull+ to download the {hosted-ems} image from the Elastic Docker registry.
52+
53+
ifeval::["{release-state}"=="unreleased"]
54+
Version {version} of {hosted-ems} has not yet been released, so no Docker image is currently available for this version.
55+
endif::[]
56+
57+
ifeval::["{release-state}"!="unreleased"]
58+
59+
["source","bash",subs="attributes"]
60+
----------------------------------
61+
docker pull {ems-docker-image}
62+
----------------------------------
63+
64+
Start {hosted-ems} and expose the default port `8080`:
65+
66+
["source","bash",subs="attributes"]
67+
----------------------------------
68+
docker run --rm --init --publish 8080:8080 \
69+
{ems-docker-image}
70+
----------------------------------
71+
72+
Once {hosted-ems} is running, follow instructions from the webpage at `localhost:8080` to define a configuration file and download the basemaps database.
73+
74+
[role="screenshot"]
75+
image::images/elastic-maps-server-instructions.png[Set-up instructions]
76+
77+
endif::[]
78+
79+
[float]
80+
[[elastic-maps-server-configuration]]
81+
==== Configuration
82+
83+
{hosted-ems} reads properties from a configuration file in YAML format that is validated on startup. The location of this file is provided by the `EMS_PATH_CONF` environment variable and defaults to `/usr/src/app/server/config/elastic-maps-server.yml`.
84+
85+
*General settings*
86+
87+
[cols="2*<"]
88+
|===
89+
| [[ems-hostname]]`hostname`
90+
| Specifies the host of the backend server. To allow remote users to connect, set the value to the IP address or DNS name of the {hosted-ems} container. *Default: _your-hostname_*. <<server-host,Equivalent {kib} setting>>.
91+
92+
| `port`
93+
| Specifies the port used by the backend server. Default: *`8080`*. <<server-port,Equivalent {kib} setting>>.
94+
95+
| `ui`
96+
| Controls the display of the status page and the layer preview. *Default: `true`*
97+
98+
| `logging.level`
99+
| Verbosity of {hosted-ems} logs. Valid values are `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `silent`. *Default: `info`*
100+
101+
| `path.planet`
102+
| Path of the basemaps database. *Default: `/usr/src/app/data/planet.mbtiles`*
103+
|===
104+
105+
106+
*{es} connection and security settings*
107+
108+
[cols="2*<"]
109+
|===
110+
111+
| `elasticsearch.host`
112+
| URL of the {es} instance to use for license validation.
113+
114+
| `elasticsearch.username` and `elasticsearch.password`
115+
| Credentials of a user with at least the `monitor` role.
116+
117+
| `elasticsearch.ssl.certificateAuthorities`
118+
| Paths to one or more PEM-encoded X.509 certificate authority (CA) certificates that make up a trusted certificate chain for {hosted-ems}. This chain is used by {hosted-ems} to establish trust when connecting to your {es} cluster. <<elasticsearch-ssl-certificateAuthorities,Equivalent {kib} setting>>.
119+
120+
| `elasticsearch.ssl.certificate` and `elasticsearch.ssl.key`, and `elasticsearch.ssl.keyPassphrase`
121+
| Optional settings that provide the paths to the PEM-format SSL certificate and key files and the key password. These files are used to verify the identity of {hosted-ems} to {es} and are required when `xpack.security.http.ssl.client_authentication` in {es} is set to `required`. <<elasticsearch-ssl-cert-key,Equivalent {kib} setting>>.
122+
123+
| `elasticsearch.ssl.verificationMode`
124+
| Controls the verification of the server certificate that {hosted-ems} receives when making an outbound SSL/TLS connection to {es}. Valid values are "`full`", "`certificate`", and "`none`". Using "`full`" performs hostname verification, using "`certificate`" skips hostname verification, and using "`none`" skips verification entirely. *Default: `full`*. <<elasticsearch-ssl-verificationMode,Equivalent {kib} setting>>.
125+
126+
|===
127+
128+
*Server security settings*
129+
130+
[cols="2*<"]
131+
|===
132+
133+
| `ssl.enabled`
134+
| Enables SSL/TLS for inbound connections to {hosted-ems}. When set to `true`, a certificate and its corresponding private key must be provided. *Default: `false`*. <<server-ssl-enabled,Equivalent {kib} setting>>.
135+
136+
| `ssl.certificateAuthorities`
137+
| Paths to one or more PEM-encoded X.509 certificate authority (CA) certificates that make up a trusted certificate chain for {hosted-ems}. This chain is used by the {hosted-ems} to establish trust when receiving inbound SSL/TLS connections from end users. <<server-ssl-certificateAuthorities,Equivalent {kib} setting>>.
138+
139+
| `ssl.key`, `ssl.certificate`, and `ssl.keyPassphrase`
140+
| Location of yor SSL key and certificate files and the password that decrypts the private key that is specified via `ssl.key`. This password is optional, as the key may not be encrypted. <<server-ssl-cert-key,Equivalent {kib} setting>>.
141+
142+
| `ssl.supportedProtocols`
143+
| An array of supported protocols with versions.
144+
Valid protocols: `TLSv1`, `TLSv1.1`, `TLSv1.2`, `TLSv1.3`. *Default: `TLSv1.1`, `TLSv1.2`, `TLSv1.3`*. <<server-ssl-supportedProtocols,Equivalent {kib} setting>>.
145+
146+
| `ssl.cipherSuites`
147+
| Details on the format, and the valid options, are available via the
148+
https://www.openssl.org/docs/man1.1.1/man1/ciphers.html#CIPHER-LIST-FORMAT[OpenSSL cipher list format documentation].
149+
*Default: `TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384, DHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA384, ECDHE-RSA-AES256-SHA256, DHE-RSA-AES256-SHA256, HIGH,!aNULL, !eNULL, !EXPORT, !DES, !RC4, !MD5, !PSK, !SRP, !CAMELLIA`*. <<server-ssl-cipherSuites,Equivalent {kib} setting>>.
150+
|===
151+
152+
[float]
153+
[[elastic-maps-server-bind-mount-config]]
154+
===== Bind-mounted configuration
155+
156+
One way to configure {hosted-ems} is to provide `elastic-maps-server.yml` via bind-mounting. With +docker-compose+, the bind-mount can be specified like this:
157+
158+
["source","yaml",subs="attributes"]
159+
--------------------------------------------
160+
version: '2'
161+
services:
162+
{hosted-ems}:
163+
image: {ems-docker-image}
164+
volumes:
165+
- ./elastic-maps-server.yml:/usr/src/app/config/elastic-maps-server.yml
166+
--------------------------------------------
167+
168+
[float]
169+
[[elastic-maps-server-envvar-config]]
170+
===== Environment variable configuration
171+
All configuration settings can be overridden by environment variables that are named with all uppercase letters and by replacing YAML periods with underscores. For example `elasticsearch.ssl.certificate` could be overridden by the environment variable `ELASTICSEARCH_SSL_CERTIFICATE`. Boolean variables must use the `true` or `false` strings.
172+
173+
WARNING: All information that you include in environment variables is visible through the `ps` command, including sensitive information.
174+
175+
These variables can be set with +docker-compose+ like this:
176+
177+
["source","yaml",subs="attributes"]
178+
----------------------------------------------------------
179+
version: '2'
180+
services:
181+
{hosted-ems}:
182+
image: {ems-docker-image}
183+
environment:
184+
ELASTICSEARCH_HOST: http://elasticsearch.example.org
185+
ELASTICSEARCH_USERNAME: 'ems'
186+
ELASTICSEARCH_PASSWORD: 'changeme'
187+
----------------------------------------------------------
188+
189+
[float]
190+
[[elastic-maps-server-data]]
191+
==== Data
192+
193+
{hosted-ems} hosts vector layer boundaries and vector tile basemaps for the entire planet. Boundaries include world countries, global administrative regions, and specific country regions. A minimal basemap is provided with {hosted-ems}. This can be used for testing environments but won't be functional for normal operations. The full basemap (around 90GB file) needs to be mounted on the Docker container for {hosted-ems} to run normally.
194+
195+
TIP: The available basemaps and boundaries can be explored from the `/maps` endpoint in a web page that is your self-managed equivalent to https://maps.elastic.co
196+
197+
198+
[float]
199+
[[elastic-maps-server-kibana]]
200+
==== Kibana configuration
201+
202+
With {hosted-ems} running, add the `map.emsUrl` configuration key in your <<settings, kibana.yml>> file pointing to the root of the service. This setting will point {kib} to request EMS basemaps and boundaries from {hosted-ems}. Typically this will be the URL to the <<ems-hostname,hostname and port>> of {hosted-ems}. For example, `map.emsUrl: https://my-ems-server:8080`.
203+
204+
205+
[float]
206+
[[elastic-maps-server-check]]
207+
==== Status check
208+
209+
{hosted-ems} periodically runs a status check that is exposed in three different forms:
210+
211+
* At the root of {hosted-ems}, a web page will render the status of the different services.
212+
* A JSON representation of {hosted-ems} status is available at the `/status` endpoint.
213+
* The Docker https://docs.docker.com/engine/reference/builder/#healthcheck[`HEALTHCHECK`] instruction is run by default and will inform about the health of the service, running a process equivalent to the `/status` endpoint.
214+
215+
IMPORTANT: {hosted-ems} won't respond to any data request if the license validation is not fulfilled.
216+
217+
218+
[float]
219+
[[elastic-maps-server-logging]]
220+
==== Logging
221+
222+
Logs are generated in {ecs-ref}[ECS JSON format] and emitted to the standard output and to `/var/log/elastic-maps-server/elastic-maps-server.log`. The server won't rotate the logs automatically but the `logrotate` tool is installed in the image. Mount `/dev/null` to the default log path if you want to disable the output to that file.
102 KB
Loading

docs/maps/index.asciidoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
:ems-docker-repo: docker.elastic.co/elastic-maps-service/elastic-maps-server-ubi8
2+
:ems-docker-image: {ems-docker-repo}:{version}
3+
:hosted-ems: Elastic Maps Server
4+
15
[role="xpack"]
26
[[maps]]
37
= Maps

docs/settings/fleet-settings.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ experimental[]
1010
You can configure `xpack.fleet` settings in your `kibana.yml`.
1111
By default, {fleet} is enabled. To use {fleet}, you also need to configure {kib} and {es} hosts.
1212

13-
See the {ingest-guide}/index.html[{fleet}] docs for more information.
13+
See the {fleet-guide}/index.html[{fleet}] docs for more information.
1414

1515
[[general-fleet-settings-kb]]
1616
==== General {fleet} settings

0 commit comments

Comments
 (0)