Skip to content

feat(import): support stage-specific pg_restore options#7690

Merged
gbartolini merged 8 commits intocloudnative-pg:mainfrom
hanshal101:allow-pg-dump-rest
Sep 25, 2025
Merged

feat(import): support stage-specific pg_restore options#7690
gbartolini merged 8 commits intocloudnative-pg:mainfrom
hanshal101:allow-pg-dump-rest

Conversation

@hanshal101
Copy link
Contributor

@hanshal101 hanshal101 commented May 29, 2025 •
edited by gbartolini
Loading

Adds support for stage-specific parallelism and flags during pg_restore in bootstrap.initdb.import. This enables finer-grained control over performance-sensitive imports by allowing users to configure separate options for:

  • predata (e.g., schema definitions)
  • data (bulk table content)
  • postdata (constraints, indexes, triggers)

Fixes: #6874
Closes: #6874

@hanshal101 hanshal101 requested a review from a team as a code owner May 29, 2025 11:02
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 29, 2025
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.22 release-1.25 release-1.26 labels May 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 29, 2025

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot bot added the enhancement 🪄 New feature or request label May 29, 2025
@hanshal101 hanshal101 force-pushed the allow-pg-dump-rest branch 2 times, most recently from 0f1179b to 8ec8d4e Compare May 29, 2025 11:05
@saolof
Copy link

saolof commented May 31, 2025 •
edited
Loading

Not a maintainer, but this PR solves exactly the problem I had when creating the issue. Thanks for working on it!

Edit: was there a preInitSql and postInitSql?

@hanshal101 hanshal101 force-pushed the allow-pg-dump-rest branch from 8ec8d4e to 95839f0 Compare June 2, 2025 13:50
@armru
Copy link
Member

armru commented Jun 4, 2025

Hello, thanks for the contribution, adding it to the review queue for 1.27

@armru armru added this to the 1.27.0 milestone Jun 4, 2025
@hanshal101 hanshal101 force-pushed the allow-pg-dump-rest branch from 95839f0 to d61fc56 Compare June 4, 2025 16:19
@hanshal101
Copy link
Contributor Author

hanshal101 commented Jun 11, 2025

Hey maintainers let me know if there's something to be added more here!

@gbartolini
Copy link
Contributor

gbartolini commented Jul 1, 2025

Hey maintainers let me know if there's something to be added more here!

Can you please integrate the documentation? https://cloudnative-pg.io/documentation/current/database_import/#customizing-pg_dump-and-pg_restore-behavior

@hanshal101 hanshal101 force-pushed the allow-pg-dump-rest branch from a7ad779 to c47b82b Compare July 1, 2025 16:21
@hanshal101 hanshal101 requested a review from jsilvela as a code owner July 1, 2025 16:21
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jul 1, 2025
@hanshal101
Copy link
Contributor Author

hanshal101 commented Jul 1, 2025

Hello @gbartolini I have updated the Documentation too for this issue.

@mnencia mnencia force-pushed the allow-pg-dump-rest branch from 99a3a50 to ae7309e Compare July 21, 2025 10:02
@gbartolini gbartolini removed this from the 1.27.0 milestone Jul 21, 2025
@gbartolini gbartolini added do not backport This PR must not be backported - it will be in the next minor release and removed backport-requested ◀️ This pull request should be backported to all supported releases labels Aug 8, 2025
@leonardoce
Copy link
Contributor

leonardoce commented Sep 22, 2025

/test

@github-actions
Copy link
Contributor

github-actions bot commented Sep 22, 2025

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/17919792455

@leonardoce
Copy link
Contributor

leonardoce commented Sep 22, 2025

/test

@github-actions
Copy link
Contributor

github-actions bot commented Sep 22, 2025

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/17920266486

@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Sep 22, 2025
@armru armru force-pushed the allow-pg-dump-rest branch 3 times, most recently from c3d0455 to f7bf4f1 Compare September 23, 2025 13:50
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 23, 2025
@armru armru force-pushed the allow-pg-dump-rest branch from 30a3e9a to 03a05c2 Compare September 23, 2025 13:55
@armru
Copy link
Member

armru commented Sep 23, 2025

/test limit=local

@github-actions
Copy link
Contributor

github-actions bot commented Sep 23, 2025

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/17948740417

@NiccoloFei
Copy link
Collaborator

NiccoloFei commented Sep 25, 2025

Using

  bootstrap:
    initdb:
      import:
        ...
        type: microservice
        pgRestorePredataOptions:
          - '--jobs=1'
        pgRestoreDataOptions:
          - '--jobs=4'
        pgRestorePostdataOptions:
          - '--jobs=2'

Output:

{
  "level": "info",
  "ts": "2025-09-25T13:01:05.646854992Z",
  "msg": "executing database importing section",
  "logging_pod": "postgresql-dump-1-import",
  "databaseName": "app",
  "section": "pre-data"
}
{
  "level": "info",
  "ts": "2025-09-25T13:01:05.646907456Z",
  "msg": "Running pg_restore",
  "logging_pod": "postgresql-dump-1-import",
  "cmd": "pg_restore",
  "options": [
    "--jobs=1",
    "-U",
    "postgres",
    "--no-owner",
    "--no-privileges",
    "--role=app",
    "-d",
    "app",
    "--section",
    "pre-data",
    "/var/lib/postgresql/data/pgdata/dumps/app.dump"
  ]
}
{
  "level": "info",
  "ts": "2025-09-25T13:01:05.67018682Z",
  "msg": "executing database importing section",
  "logging_pod": "postgresql-dump-1-import",
  "databaseName": "app",
  "section": "data"
}
{
  "level": "info",
  "ts": "2025-09-25T13:01:05.670219712Z",
  "msg": "Running pg_restore",
  "logging_pod": "postgresql-dump-1-import",
  "cmd": "pg_restore",
  "options": [
    "--jobs=4",
    "-U",
    "postgres",
    "--no-owner",
    "--no-privileges",
    "--role=app",
    "-d",
    "app",
    "--section",
    "data",
    "/var/lib/postgresql/data/pgdata/dumps/app.dump"
  ]
}
{
  "level": "info",
  "ts": "2025-09-25T13:01:05.70045799Z",
  "msg": "executing database importing section",
  "logging_pod": "postgresql-dump-1-import",
  "databaseName": "app",
  "section": "post-data"
}
{
  "level": "info",
  "ts": "2025-09-25T13:01:05.700509499Z",
  "msg": "Running pg_restore",
  "logging_pod": "postgresql-dump-1-import",
  "cmd": "pg_restore",
  "options": [
    "--jobs=2",
    "-U",
    "postgres",
    "--no-owner",
    "--no-privileges",
    "--role=app",
    "-d",
    "app",
    "--section",
    "post-data",
    "/var/lib/postgresql/data/pgdata/dumps/app.dump"
  ]
}

hanshal101 and others added 8 commits September 25, 2025 15:12
@hanshal101 @NiccoloFei
feat: allow extra pg_dump/pg_restore settings to apply only to predat…
703a8f2 
…a/postdata

Signed-off-by: hanshal101 <hanshalmehta10@gmail.com>
@hanshal101 @NiccoloFei
feat: add docs
1e97985 
Signed-off-by: hanshal101 <hanshalmehta10@gmail.com>
@gbartolini @NiccoloFei
docs: basic review of documentation
f08dd8a 
Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
@leonardoce @NiccoloFei
chore: update CRD manifest
9a49d32 
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
@leonardoce @NiccoloFei
chore: review and refactor
2ff519a 
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
@armru @NiccoloFei
chore: typo
9bd940e 
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@armru @NiccoloFei
refactor: rename PgRestoreSectionName to section given that getSe…
9f8de29 
…ctionsToExecute is used also for pg_dump

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@NiccoloFei
docs: minor fixes
9cb82a2 
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
@gbartolini gbartolini changed the title feat: allow extra pg_dump/pg_restore settings on predata/postdata feat(import): support stage-specific pg_restore options Sep 25, 2025
@gbartolini gbartolini merged commit 21e5bfe into cloudnative-pg:main Sep 25, 2025
30 checks passed
rossigee pushed a commit to rossigee/cloudnative-pg that referenced this pull request Oct 2, 2025
@hanshal101 @gbartolini
@leonardoce @armru @NiccoloFei @rossigee
feat(import): support stage-specific pg_restore options (cloudnativ…
bb4e251 
…e-pg#7690)

Adds support for stage-specific parallelism and flags during
`pg_restore` in `bootstrap.initdb.import`. This enables finer-grained
control over performance-sensitive imports by allowing users to
configure separate options for:

- predata (e.g., schema definitions)
- data (bulk table content)
- postdata (constraints, indexes, triggers)

Fixes: cloudnative-pg#6874 
Closes: cloudnative-pg#6874

Signed-off-by: hanshal101 <hanshalmehta10@gmail.com>
Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Co-authored-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
THE-BRAHMA pushed a commit to THE-BRAHMA/cloudnative-pg that referenced this pull request Oct 30, 2025
@hanshal101 @gbartolini
@leonardoce @armru @NiccoloFei @THE-BRAHMA
feat(import): support stage-specific pg_restore options (cloudnativ…
490aa63 
…e-pg#7690)

Adds support for stage-specific parallelism and flags during
`pg_restore` in `bootstrap.initdb.import`. This enables finer-grained
control over performance-sensitive imports by allowing users to
configure separate options for:

- predata (e.g., schema definitions)
- data (bulk table content)
- postdata (constraints, indexes, triggers)

Fixes: cloudnative-pg#6874
Closes: cloudnative-pg#6874

Signed-off-by: hanshal101 <hanshalmehta10@gmail.com>
Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Co-authored-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Signed-off-by: theBrahma <office.utpal.brahma@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leonardoce leonardoce leonardoce approved these changes

@NiccoloFei NiccoloFei NiccoloFei approved these changes

@armru armru armru approved these changes

@jsilvela jsilvela Awaiting requested review from jsilvela jsilvela is a code owner

Assignees

No one assigned

Labels

do not backport This PR must not be backported - it will be in the next minor release enhancement 🪄 New feature or request lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged priority:low size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Allow extra pg_dump/pg_restore settings to apply only to predata/postdata

7 participants

@hanshal101 @saolof @armru @gbartolini @leonardoce @NiccoloFei @cnpg-bot