Skip to content

Doc: Update upgrade info for 8.0 in main #13705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
karenzone merged 4 commits into from
Feb 9, 2022
Merged

Doc: Update upgrade info for 8.0 in main #13705

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fce1f8f
Doc: Update upgrade info for 8.0
karenzone Feb 3, 2022
6a2e10f
More cleanup and changes to PQ upgrade info
karenzone Feb 3, 2022
3f9adb2
Merge branch 'main' into upgrade
karenzone Feb 9, 2022
a18ee1d
Incorporate review comments and fix links
karenzone Feb 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/static/breaking-changes-pre63.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@

If you are upgrading from Logstash 6.2.x or any earlier version (including 5.x)
and have the persistent queue enabled, we strongly recommend that you drain or
delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
delete the persistent queue before you upgrade. See <<draining-pqs>>
for information and instructions.
Copy link
Copy Markdown
Contributor Author

@karenzone karenzone Feb 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this info is in Breaking Changes, not Upgrade. If we decide to remove it post-release, we can do that. For now, I changed the link to avoid breaking the doc build by pointing to content that has been removed.


146 changes: 43 additions & 103 deletions docs/static/upgrading.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,54 +24,56 @@ See the following topics for information about upgrading Logstash:
* <<upgrading-using-package-managers>>
* <<upgrading-using-direct-download>>
* <<upgrading-minor-versions>>
* <<upgrading-logstash-7.0>>
* <<upgrading-logstash-pqs>>
* <<upgrading-logstash-8.0>>

[float]
==== When to Upgrade
[discrete]
==== When to upgrade

Fresh installations can and should start with the same version across the Elastic Stack.

Elasticsearch 7.0 does not require Logstash 7.0. An Elasticsearch 7.0 cluster
Elasticsearch 8.0 does not require Logstash 8.0. An Elasticsearch 8.0 cluster
will happily receive data from earlier versions of Logstash via the default
HTTP communication layer. This provides some flexibility to decide when to
upgrade Logstash relative to an Elasticsearch upgrade. It may or may not be
convenient for you to upgrade them together, and it is not required to be done
at the same time as long as Elasticsearch is upgraded first.

You should upgrade in a timely manner to get the performance improvements that
come with Logstash 7.0, but do so in the way that makes the most sense for your
come with Logstash 8.0, but do so in the way that makes the most sense for your
environment.

[float]
==== When Not to Upgrade
[discrete]
==== When not to upgrade

If any Logstash plugin that you require is not compatible with Logstash 7.0, then you should wait until it is ready
If any Logstash plugin that you require is not compatible with Logstash 8.0, then you should wait until it is ready
before upgrading.

Although we make great efforts to ensure compatibility, Logstash 7.0 is not completely backwards compatible. As noted
in the Elastic Stack upgrade guide, Logstash 7.0 should not be upgraded before Elasticsearch 7.0. This is both
practical and because some Logstash 7.0 plugins may attempt to use features of Elasticsearch 7.0 that did not exist
in earlier versions. For example, if you attempt to send the 7.x template to a cluster before Elasticsearch 7.0, then
all indexing likely fail. If you use your own custom template with Logstash,
then this issue can be ignored.
Although we make great efforts to ensure compatibility, Logstash 8.0 is not completely backwards compatible.
As noted in the Elastic Stack upgrade guide, you should not upgrade Logstash 8.0 before you upgrade Elasticsearch 8.0.
This is both
practical and because some Logstash 8.0 plugins may attempt to use features of Elasticsearch 8.0 that did not exist
in earlier versions.

For example, if you attempt to send the 8.x template to a cluster before
Elasticsearch 8.0, then all indexing likely fail.
If you use your own custom template with Logstash, then this issue can be ignored.


[[upgrading-using-package-managers]]
=== Upgrading Using Package Managers
=== Upgrading using package managers

This procedure uses <<package-repositories,package managers>> to upgrade Logstash.

1. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
2. Using the directions in the <<package-repositories>> section, update your repository
links to point to the 7.x repositories.
3. Run the `apt-get upgrade logstash` or `yum update logstash` command as appropriate for your operating system.
4. Test your configuration file with the `logstash --config.test_and_exit -f <configuration-file>` command. Configuration options for
some Logstash plugins have changed in the 7.x release.
5. Restart your Logstash pipeline after you have updated your configuration file.
. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
. Using the directions in the <<package-repositories>> section, update your repository
links to point to the 8.x repositories.
. Run the `apt-get upgrade logstash` or `yum update logstash` command as appropriate for your operating system.
. Test your configuration file with the `logstash --config.test_and_exit -f <configuration-file>` command. Configuration options for
some Logstash plugins have changed in the 8.x release.
. Restart your Logstash pipeline after you have updated your configuration file.

[[upgrading-using-direct-download]]
=== Upgrading Using a Direct Download
=== Upgrading using a direct download
Copy link
Copy Markdown
Member

@andsel andsel Feb 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a side note, the PR #13684 touches the same list, in case we want to have a unique PR

Copy link
Copy Markdown
Contributor Author

@karenzone karenzone Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged before this PR


This procedure downloads the relevant Logstash binaries directly from Elastic.

Expand All @@ -89,18 +91,18 @@ some Logstash plugins have changed.
[[upgrading-minor-versions]]
=== Upgrading between minor versions

As a general rule, you can upgrade between minor versions (for example, 7.x to
7.y, where x < y) by simply installing the new release and restarting {ls}.
As a general rule, you can upgrade between minor versions (for example, 8.x to
8.y, where x < y) by simply installing the new release and restarting {ls}.
{ls} typically maintains backwards compatibility for configuration
settings and exported fields. Please review the
<<releasenotes,release notes>> for potential exceptions.

Upgrading between non-consecutive major versions (5.x to 7.x, for example) is not
supported.
Upgrading between non-consecutive major versions (6.x to 8.x, for example) is
not supported.


[[upgrading-logstash-7.0]]
=== Upgrading Logstash to 7.0
[[upgrading-logstash-8.0]]
=== Upgrading Logstash to 8.0

Before upgrading Logstash:

Expand All @@ -109,89 +111,27 @@ Before upgrading Logstash:
+
There you can find info on these topics and more:

** <<java-exec-default,Java execution engine enabled by default>>
** <<field-ref-strict,Field parser is more strict and how that affects processing>>
** <<beats-ecs,Beats conforms to the Elastic Common Schema (ECS) and how that impacts {ls}>>
** <<bc-ecs-compatibility,Elastic Common Schema (ECS) compatibility on by default>>
** <<security-on-8.0,Secure communication with Elasticsearch>>
** <<bc-field-ref-parser,Field parser is more strict>>


If you are installing Logstash with other components in the Elastic Stack, also see the
{stack-ref}/index.html[Elastic Stack installation and upgrade documentation].

NOTE: Upgrading between non-consecutive major versions (5.x to 7.x, for example) is not
supported. We recommend that you upgrade to 6.x, and then upgrade to 7.x.
NOTE: Upgrading between non-consecutive major versions (6.x to 8.x, for example) is not supported.
We recommend that you upgrade to {prev-major-last}, and then upgrade to 8.0.

[float]
[[upgrade-to-6.8-rec]]
==== Upgrade to {ls} 6.8 before upgrading to 7.0
[discrete]
[[upgrade-to-previous]]
==== Upgrade to {ls} {prev-major-last} before upgrading to 8.0

If you haven't already, upgrade to version 6.8 before you upgrade to 7.0. If
If you haven't already, upgrade to version {prev-major-last} before you upgrade to 8.0. If
you're using other products in the {stack}, upgrade {ls} as part of the
{stack-ref}/upgrading-elastic-stack.html[{stack} upgrade process].

TIP: Upgrading to {ls} 6.8 will give you a head-start on new 7.0 features, including
the java execution engine and the strict field reference parser, while you're still running 6.x.
TIP: Upgrading to {ls} {prev-major-last} gives you a head-start on new 8.0 features.
This step helps reduce risk and makes roll backs easier if you hit
a snag.

//TO DO: Add links [[field-ref-strict]] and [[java-exec-default]] after upgrade docs are merged

Upgrading to 6.8 is required because the {es} index template was modified to
be compatible with {es} 7.0 (the `_type` setting changed from `doc` to `_doc`).


[[upgrading-logstash-pqs]]
=== Upgrading with the Persistent Queue Enabled

If you have the persistent queue (PQ) enabled, please read the section that applies
for your upgrade scenario.

* If you are upgrading from version 6.2.x or earlier, we recommend that you
<<drain-pq,drain the persistent queue>> before you upgrade.

* If you are upgrading from version 6.3.0 or later, see
<<upgrading-logstash-pqs-6.3>> for information.

[float]
[[drain-pq]]
==== Drain the Persistent Queue (version 6.2.x and earlier)

The following applies only if you are upgrading from Logstash version 6.2.x or
earlier with the persistent queue (PQ) enabled.

We strive to maintain backward compatibility within a given major release.
Serialization issues in Logstash 6.2.x and earlier required us to break
that compatibility in version 6.3.0 to ensure correctness of operation. For more
technical details, please check our tracking github issue for this
matter, https://github.com/elastic/logstash/issues/9494[#9494].

We strongly recommend that you drain or delete
the persistent queue before you upgrade from version 6.2.x and earlier.

To drain the queue:

. In the logstash.yml file, set `queue.drain: true`.
. Restart Logstash for this setting to take effect.
. Shutdown Logstash (using CTRL+C or SIGTERM), and wait for the queue to empty.

When the queue is empty:

. Complete the upgrade.
. Restart Logstash.

We have resolved issues with data incompatibilities for version 6.3 and later.
These steps won’t be required for future upgrades.

[float]
[[upgrading-logstash-pqs-6.3]]
==== Upgrading from version 6.3 (and later) with Persistent Queues Enabled

Upgrading Logstash with persistent queues enabled is supported. The persistent
queue directory is self-contained and can be read by a new Logstash instance
running the same pipeline. You can safely shut down the original Logstash
instance, spin up a new instance, and set `path.queue` in the `logstash.yml`
<<logstash-settings-file,settings file>> to point to the original queue directory.
You can also use a mounted drive to make this workflow easier.

Keep in mind that only one Logstash instance can write to `path.queue`. You
cannot have the original instance and the new instance writing to the queue at
the same time.