Skip to content

fix(targetTime): clarify timestamp formats and UTC interpretation#8937

Merged
mnencia merged 5 commits intocloudnative-pg:mainfrom
pchovelon:dev-8928
Dec 30, 2025
Merged

fix(targetTime): clarify timestamp formats and UTC interpretation#8937
mnencia merged 5 commits intocloudnative-pg:mainfrom
pchovelon:dev-8928

Conversation

@pchovelon
Copy link
Contributor

@pchovelon pchovelon commented Oct 24, 2025

Update the machinery dependency to v0.3.3 to ensure timestamps without timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC.

The documentation has been updated to clarify that targetTime accepts both RFC 3339 and PostgreSQL timestamp formats.

Closes #8928

Related: cloudnative-pg/machinery#167

@pchovelon pchovelon requested review from a team and jsilvela as code owners October 24, 2025 07:31
@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Oct 24, 2025
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.26 release-1.27 labels Oct 24, 2025
@github-actions
Copy link
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 24, 2025
@mnencia mnencia changed the title Fix documentation. Wrong targetTime format. docs: fix dwrong targetTime format Oct 24, 2025
@mnencia mnencia changed the title docs: fix dwrong targetTime format docs: fix wrong targetTime format Oct 24, 2025
@jsilvela
Copy link
Collaborator

Was looking at this, and have a concern.

The code still says RFC3339. See cluster_types.go

type RecoveryTarget struct {
	<------->
	// The target time as a timestamp in the RFC3339 standard
	// +optional
	TargetTime string `json:"targetTime,omitempty"`

And in the machinery repo, in time.go, the below function takes RFC3339, RFC3339Micro, or timestamp

func ConvertToPostgresFormat(timestamp string) string {

So, if the documentation is indeed wrong, and only timestamps are to be allowed, shouldn't the code change too?
The doc and code are definitely out of sync.

Copy link
Collaborator

@jsilvela jsilvela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue with the targetTime in the document is the missing Z at the end.
The targetTime does admit RFC3339, RFC3339Micro, and timestamp.
That said, the code for handling is a bit convoluted and could use some updating.

apiGroup: snapshot.storage.k8s.io
recoveryTarget:
targetTime: "2023-07-06T08:00:39"
targetTime: "2023-06-11 08:00:39.00000+02"
Copy link
Collaborator

@jsilvela jsilvela Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Been digging a bit in the code.
The problem with the existing targetTime in the doc, "2023-07-06T08:00:39", is that it's not valid RFC3339. To have a zero timezone offset, you need a final "Z".
This worked in a unit test.

recoveryTarget := RecoveryTarget{
	TargetTime: "2023-07-06T08:00:39Z",
}

	Expect(recoveryTarget.BuildPostgresOptions()).To(ContainSubstring("recovery_target_time = '2023-07-06 08:00:39.000000Z'"))

@dosubot dosubot bot removed the lgtm This PR has been approved by a maintainer label Oct 27, 2025
targetTime
: Time stamp up to which recovery proceeds, expressed in
[RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format.
: Time stamp up to which recovery proceeds, expressed in the form of a timestamp.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to add in the review. Timestamps are admitted, definitely, but so is RFC3339, see my previous comment on the missing Z.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See test below :)

@jsilvela
Copy link
Collaborator

jsilvela commented Oct 27, 2025

And, since @mnencia approved previously: I'd like your take on this.
There is a problem here in the code in that we have asymmetry.
The webhook admits the targetTime formated in as in the documentation ("2023-07-06T08:00:39", which is not valid RFC3339 because of the missing Z), but the processing will not generate a proper Postgres timestamp.

The webhook uses func ParseTargetTime at /cloudnative-pg/machinery@v0.3.1/pkg/types/time.go, which specifically admits the above format.

BUT, in func ConvertToPostgresFormat /cloudnative-pg/machinery@v0.3.1/pkg/postgres/time/time.go, that format is passed through, and it is NOT a valid Postgres format, whereas RFC3339 and RFC3339Micro are regenerated into Postgres format.

Why do we have the special format allowed in ParseTargetTime?

@pchovelon
Copy link
Contributor Author

pchovelon commented Oct 27, 2025

After your comments @jsilvela, I tried again an unit test to confirm there is an issue, event with Z at the end of timestamp.
Here are steps :

First create a cluster with backup section using plugin :

---
apiVersion: v1
kind: Secret
metadata:
  name: minio-creds
type: Opaque
data:
  ACCESS_KEY_ID: bWluaW9hZG1pbg==
  ACCESS_REGION: ZnItcGFy
  ACCESS_SECRET_KEY: bWluaW9hZG1pbg==
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-alice
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:17.5
  instances: 1
  storage:
    size: 1Gi
  # Backup properties
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters:
      barmanObjectName: minio-store
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: minio-store
spec:
  configuration:
    destinationPath: s3://demo/pierrick
    endpointURL: http://172.18.0.3:9000
    s3Credentials:
      accessKeyId:
        name: minio-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: minio-creds
        key: ACCESS_SECRET_KEY
      region:
        name: minio-creds
        key: ACCESS_REGION

Create backup
`kubectl cnpg backup cluster-alice --method=plugin --plugin-name=barman-cloud.cloudnative-pg.io

k describe backups.postgresql.cnpg.io cluster-alice-20251027172908
...
  Plugin Metadata:
    Cluster UID:   def9ae28-9aea-4a99-833e-e05a61c03901
    Display Name:  BarmanCloudInstance
    Name:          barman-cloud.cloudnative-pg.io
    Plugin Name:   barman-cloud.cloudnative-pg.io
    Timeline:      1
    Version:       0.7.0
  Started At:      2025-10-27T16:29:09Z
  Stopped At:      2025-10-27T16:29:10Z
Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Starting   10m   cloudnative-pg-backup  Starting backup for cluster cluster-alice
  Normal  Starting   10m   local-webserver        Backup started
  Normal  Completed  10m   local-webserver        Backup completed

Create a cluster with bootstrap restore method

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-bob
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:17.5
  instances: 1
  storage:
    size: 1Gi
  bootstrap:
    recovery:
      source: backup-cluster-alice
        targetTime: "2025-10-27T16:30:10Z"
  externalClusters:
    - name: backup-cluster-alice
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: minio-store
          serverName: cluster-alice

The bob cluster start with a recovery Pod :

{
  "level": "info",
  "ts": "2025-10-27T16:32:59.56105952Z",
  "msg": "Starting webserver",
  "logging_pod": "cluster-bob-1-full-recovery",
  "address": "localhost:8010",
  "hasTLS": false
}

... but get this error :

{
  "level": "info",
  "ts": "2025-10-27T16:33:03.221008034Z",
  "msg": "Starting up instance",
  "logging_pod": "cluster-bob-1-full-recovery",
  "pgdata": "/var/lib/postgresql/data/pgdata",
  "options": [
    "start",
    "-w",
    "-D",
    "/var/lib/postgresql/data/pgdata",
    "-o",
    "-c port=5432 -c unix_socket_directories=/controller/run",
    "-t 40000000",
    "-o",
    "-c listen_addresses='127.0.0.1'"
  ]
}
{
  "level": "info",
  "ts": "2025-10-27T16:33:03.238768322Z",
  "logger": "pg_ctl",
  "msg": "waiting for server to start....2025-10-27 16:33:03.238 UTC [46] LOG:  invalid value for parameter \"recovery_target_time\": \"2025-10-27 16:30:10.000000Z\"",
  "pipe": "stdout",
  "logging_pod": "cluster-bob-1-full-recovery"
}
{
  "level": "info",
  "ts": "2025-10-27T16:33:03.238788644Z",
  "logger": "pg_ctl",
  "msg": "2025-10-27 16:33:03.238 UTC [46] FATAL:  configuration file \"/var/lib/postgresql/data/pgdata/custom.conf\" contains errors",
  "pipe": "stdout",
  "logging_pod": "cluster-bob-1-full-recovery"
}
{
  "level": "info",
  "ts": "2025-10-27T16:33:03.33110249Z",
  "logger": "pg_ctl",
  "msg": "pg_ctl: could not start server",
  "pipe": "stderr",
  "logging_pod": "cluster-bob-1-full-recovery"
}

LOG: invalid value for parameter \"recovery_target_time\": \"2025-10-27 16:30:10.000000Z\""

@jsilvela
Copy link
Collaborator

@pchovelon huh! The conversion from RFC3339 to postgres was partial. I was expecting full or pass-through, from the code ... I'll try to zero on this tomorrow.

@jsilvela
Copy link
Collaborator

OK, @pchovelon , pretty stumped.
The conversion above was not partial, but complete and correct. The final Z for UTC is in both the RFC3339 format, and ISO 8601, which Postgres is compatible with.

And, the string you show is a valid Postgres timestamp:

select timestamp with time zone  '2025-10-27 16:30:10.000000Z';
      timestamptz       
------------------------
 2025-10-27 16:30:10+00

So, may be a specific issue with recovery_target_time.
I'm getting more confused with time (pun intended).

@jsilvela
Copy link
Collaborator

According to postgres, if it's valid as a timestamp with time zone, it ought to work. 🤔

The value of this parameter is a time stamp in the same format accepted by the timestamp with time zone data type

https://www.postgresql.org/docs/17/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET

@jsilvela
Copy link
Collaborator

Howdy @pchovelon . I've made some experiments.
The problem is with times formatted with a trailing Z (those in UTC zone).
In the Postgres doc for recovery_target_time you can see

The value of this parameter is a time stamp in the same format accepted by the timestamp with time zone data type, except that you cannot use a time zone abbreviation

If the RFC3339 timestamp input in the manifest is instead converted to use +00 for UTC, instead of Z, PITR works fine.

Your doc fix, IMO, while relevant, should be changed:

  • on one hand, if we say we accept RFC3339 in the manifest, let's make it work rather than give up on it.
  • on the other hand, the docstrings in the code make it seem like ONLY RFC3339 is supported, which is not accurate.

I've opened cloudnative-pg/machinery#167 to fix the conversion function.
IMO while the code is easy and I'll get it out today, this requires pondering from the maintainers.

@pchovelon
Copy link
Contributor Author

pchovelon commented Oct 30, 2025

Howdy @pchovelon . I've made some experiments.

Great, thank you @jsilvela . Running after time here ... I try to answer asap.

Copy link
Contributor Author

@pchovelon pchovelon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change doc to include RFC3339 as the bug will be fixed with #168

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 24, 2025
@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Dec 24, 2025
@armru
Copy link
Member

armru commented Dec 29, 2025

/test

@github-actions
Copy link
Contributor

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20577814711

pchovelon and others added 3 commits December 29, 2025 17:49
Signed-off-by: Pierrick Chovelon <pierrick.chovelon@dalibo.com>
Co-authored-by: Jaime Silvela <jaime.silvela@mailfence.com>
Signed-off-by: Pierrick <139142330+pchovelon@users.noreply.github.com>
Update the machinery dependency to include the fix that ensures
RFC3339-like timestamps without timezone (e.g., "2023-07-06T08:00:39")
are interpreted as UTC.

The documentation has been updated to clarify that RFC 3339 timestamps
without an explicit timezone suffix are interpreted as UTC.

Related: cloudnative-pg/machinery#167
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
@mnencia mnencia dismissed jsilvela’s stale review December 30, 2025 08:34

All the changes requested by Jaime have been implemented.

Update documentation and API reference to clarify that targetTime accepts
both RFC3339 and PostgreSQL timestamp formats. Document that timestamps
without an explicit timezone are interpreted as UTC.

Add link to PostgreSQL documentation for recovery_target_time and include
a warning recommending users to always specify an explicit timezone to
avoid ambiguity.

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Dec 30, 2025
@mnencia mnencia changed the title fix(targetTime): treat RFC3339-like timestamps without timezone as UTC fix(targetTime): clarify timestamp formats and UTC interpretation Dec 30, 2025
@mnencia mnencia merged commit 1a64853 into cloudnative-pg:main Dec 30, 2025
39 checks passed
cnpg-bot pushed a commit that referenced this pull request Dec 30, 2025
)

Update the machinery dependency to v0.3.3 to ensure timestamps without
timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC.

The documentation has been updated to clarify that targetTime accepts
both RFC 3339 and PostgreSQL timestamp formats.

Closes #8928

Related: cloudnative-pg/machinery#167

Signed-off-by: Pierrick Chovelon <pierrick.chovelon@dalibo.com>
Signed-off-by: Pierrick <139142330+pchovelon@users.noreply.github.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Jaime Silvela <jaime.silvela@mailfence.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
(cherry picked from commit 1a64853)
cnpg-bot pushed a commit that referenced this pull request Dec 30, 2025
)

Update the machinery dependency to v0.3.3 to ensure timestamps without
timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC.

The documentation has been updated to clarify that targetTime accepts
both RFC 3339 and PostgreSQL timestamp formats.

Closes #8928

Related: cloudnative-pg/machinery#167

Signed-off-by: Pierrick Chovelon <pierrick.chovelon@dalibo.com>
Signed-off-by: Pierrick <139142330+pchovelon@users.noreply.github.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Jaime Silvela <jaime.silvela@mailfence.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
(cherry picked from commit 1a64853)
mnencia added a commit that referenced this pull request Dec 30, 2025
)

Update the machinery dependency to v0.3.3 to ensure timestamps without
timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC.

The documentation has been updated to clarify that targetTime accepts
both RFC 3339 and PostgreSQL timestamp formats.

Closes #8928

Related: cloudnative-pg/machinery#167

Signed-off-by: Pierrick Chovelon <pierrick.chovelon@dalibo.com>
Signed-off-by: Pierrick <139142330+pchovelon@users.noreply.github.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Jaime Silvela <jaime.silvela@mailfence.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
(cherry picked from commit 1a64853)
mnencia pushed a commit that referenced this pull request Feb 17, 2026
…39 format (#8981)

Exercises the machinery fix (cloudnative-pg/machinery#168) that allows
RFC3339 timestamps in recovery_target_time. The existing PITR snapshot
test is extended to also restore with an RFC3339-formatted targetTime,
ensuring both formats work end-to-end.

Ref: #8937,
cloudnative-pg/machinery#167

Signed-off-by: Jaime Silvela <jaime.silvela@mailfence.com>
cnpg-bot pushed a commit to cloudnative-pg/artifacts that referenced this pull request Feb 18, 2026
…39 format (#8981)%0A%0AExercises the machinery fix (cloudnative-pg/machinery#168) that allows%0ARFC3339 timestamps in recovery_target_time. The existing PITR snapshot%0Atest is extended to also restore with an RFC3339-formatted targetTime,%0Aensuring both formats work end-to-end.%0A%0ARef: cloudnative-pg/cloudnative-pg#8937: Jaime Silvela <jaime.silvela@mailfence.com>

cloudnative-pg/cloudnative-pg@refs/heads/main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases documentation 📖 Improvements or additions to documentation lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.25 release-1.27 release-1.28 size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docs]: Incorrect syntax for targetTime in documented example

6 participants