fix(walrestore): prevent replicas from downloading future timeline history files by mnencia · Pull Request #9650 · cloudnative-pg/cloudnative-pg

mnencia · 2026-01-08T16:32:25Z

Replicas can crash-loop when orphaned "future timeline" .history files
exist in the WAL archive. This can occur during split-brain scenarios
or other conditions where timeline history files are created for timelines
that the cluster never officially adopts.

This fix adds validation during WAL restore to prevent replicas from
downloading timeline history files with timeline IDs greater than the
cluster's current timeline. Primary instances retain full access.

The validation works by:

Parsing timeline ID from .history filenames (e.g., 00000022.history)
Checking if the instance is a primary or replica
For replicas, rejecting files where fileTimeline > clusterTimeline
Returning "file not found" to PostgreSQL for rejected files

This prevents PostgreSQL from ever seeing the problematic history file,
allowing normal recovery to proceed. Combined with PR #9637 (auto-recovery
via re-cloning), this provides complete coverage of timeline divergence
scenarios.

Closes #4188

github-actions · 2026-01-08T16:32:36Z

❗ By default, the pull request is configured to backport to all release branches.

To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

mnencia · 2026-01-08T16:39:13Z

/test

github-actions · 2026-01-08T16:39:23Z

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20824279808

mnencia · 2026-01-09T14:20:00Z

/test

github-actions · 2026-01-09T14:20:12Z

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20854827166

armru · 2026-01-12T11:32:41Z

/test

github-actions · 2026-01-12T11:32:50Z

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20917830755

tests/e2e/backup_restore_minio_test.go

mnencia · 2026-01-12T18:35:31Z

/test

github-actions · 2026-01-12T18:35:41Z

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20930738286

…story files Replicas can crash-loop when orphaned "future timeline" .history files exist in the WAL archive. This can occur during split-brain scenarios or other conditions where timeline history files are created for timelines that the cluster never officially adopts. This fix adds validation during WAL restore to prevent replicas from downloading timeline history files with timeline IDs greater than the cluster's current timeline. Primary instances retain full access. The validation works by: - Parsing timeline ID from .history filenames (e.g., 00000022.history) - Checking if the instance is a primary or replica - For replicas, rejecting files where fileTimeline > clusterTimeline - Returning "file not found" to PostgreSQL for rejected files This prevents PostgreSQL from ever seeing the problematic history file, allowing normal recovery to proceed. Combined with PR #9637 (auto-recovery via re-cloning), this provides complete coverage of timeline divergence scenarios. Closes #4188 Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Add comprehensive unit tests for the timeline history file validation logic. Tests cover regular WAL files, invalid filenames, primary behavior, and replica behavior with current, past, and future timelines. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Add E2E test demonstrating protection against future timeline history files through a backup and restore scenario. The test creates two clusters sharing a WAL archive, where cluster 2 creates timeline 2 history files. When cluster 1 scales up, the new replica successfully joins without crash-looping, validating the protection works correctly. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Move the timeline divergence test into its own Context block with its own BeforeAll hook that creates a dedicated namespace and MinIO setup. This allows the test to run independently without requiring the main backup test cluster to remain running, reducing peak resource usage from 5 to 3 PostgreSQL instances during timeline test execution. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

…story files (#9650) Replicas can crash-loop when orphaned "future timeline" .history files exist in the WAL archive. This can occur during split-brain scenarios or other conditions where timeline history files are created for timelines that the cluster never officially adopts. This fix adds validation during WAL restore to prevent replicas from downloading timeline history files with timeline IDs greater than the cluster's current timeline. Primary instances retain full access. The validation works by: - Parsing timeline ID from .history filenames (e.g., 00000022.history) - Checking if the instance is a primary or replica - For replicas, rejecting files where fileTimeline > clusterTimeline - Returning "file not found" to PostgreSQL for rejected files This prevents PostgreSQL from ever seeing the problematic history file, allowing normal recovery to proceed. Combined with PR #9637 (auto-recovery via re-cloning), this provides complete coverage of timeline divergence scenarios. Closes #4188 Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> (cherry picked from commit ef73994)

Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit 1044c8e) (cherry picked from commit d0e801ee5392465d8629056c79e58c32b215dcff)

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit 1044c8e) (cherry picked from commit d0e801ee5392465d8629056c79e58c32b215dcff) (cherry picked from commit 9e2e469)

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit 1044c8e)

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit 1044c8e) (cherry picked from commit d0e801ee5392465d8629056c79e58c32b215dcff)

…9849) Move timeline history file validation to execute before any WAL restore attempt (plugin or in-tree) rather than only for in-tree restores. This prevents replicas from downloading timeline history files with timeline IDs higher than the cluster's current timeline when plugins handle WAL restore. The timeline protection added in #9650 was only applied to in-tree WAL restore, but not to plugin-based restore. This allowed the protection to be completely bypassed when using plugins, causing replicas to download future timeline history files and fail with timeline mismatch errors. Fixes the timeline protection introduced in ef73994. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit 1044c8e) (cherry picked from commit d0e801ee5392465d8629056c79e58c32b215dcff) (cherry picked from commit 9e2e469)

mnencia requested a review from a team as a code owner January 8, 2026 16:32

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jan 8, 2026

cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.27 release-1.28 labels Jan 8, 2026

dosubot bot added the bug 🐛 Something isn't working label Jan 8, 2026

mnencia force-pushed the dev/4188 branch 3 times, most recently from 3ffa226 to 34bf9be Compare January 8, 2026 17:17

cnpg-bot added the ok to merge 👌 This PR can be merged label Jan 8, 2026

mnencia force-pushed the dev/4188 branch from 34bf9be to 705c15b Compare January 9, 2026 13:01

mnencia requested review from NiccoloFei, jsilvela and litaocdl as code owners January 9, 2026 13:01

mnencia force-pushed the dev/4188 branch 3 times, most recently from b60ebdc to 6e3957e Compare January 9, 2026 14:11

armru force-pushed the dev/4188 branch from 6e3957e to 711997c Compare January 12, 2026 11:24

armru approved these changes Jan 12, 2026

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jan 12, 2026

mnencia commented Jan 12, 2026

View reviewed changes

tests/e2e/backup_restore_minio_test.go Show resolved Hide resolved

mnencia force-pushed the dev/4188 branch from f3f45f1 to 4f126f8 Compare January 12, 2026 18:35

gbartolini force-pushed the dev/4188 branch from 4f126f8 to 60ca747 Compare January 14, 2026 22:46

jbattiato approved these changes Jan 19, 2026

View reviewed changes

mnencia force-pushed the dev/4188 branch from e0ec0c9 to dbb8c9f Compare January 19, 2026 16:24

mnencia and others added 6 commits January 20, 2026 10:16

test(e2e): avoid hardcoding names

dc4b76a

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

test(e2e): follow the resource deletion pattern

572b73c

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

mnencia force-pushed the dev/4188 branch from dbb8c9f to 56d115d Compare January 20, 2026 09:16

mnencia merged commit ef73994 into main Jan 20, 2026
34 checks passed

mnencia deleted the dev/4188 branch January 20, 2026 13:02

ardentperf mentioned this pull request Jan 29, 2026

[Bug]: Replica does not come up: checkpoint not found in timeline #4990

Open

4 tasks

nrlulz mentioned this pull request Jan 29, 2026

[Bug]: Problem upgrading on a restored cluster. #9764

Closed

4 tasks

mnencia mentioned this pull request Feb 3, 2026

fix(walrestore): apply timeline validation before plugin WAL restore #9849

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(walrestore): prevent replicas from downloading future timeline history files#9650

fix(walrestore): prevent replicas from downloading future timeline history files#9650
mnencia merged 6 commits intomainfrom
dev/4188

mnencia commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

mnencia commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

mnencia commented Jan 9, 2026

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

armru commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

Uh oh!

mnencia commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mnencia commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

mnencia commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

mnencia commented Jan 9, 2026

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

armru commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

Uh oh!

mnencia commented Jan 12, 2026

Uh oh!

github-actions bot commented Jan 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants