Skip to content

Handle longer strings for netCDF history, remark and title#6084

Merged
PaulWessel merged 5 commits intomasterfrom
longer-title-remark-history
Dec 4, 2021
Merged

Handle longer strings for netCDF history, remark and title#6084
PaulWessel merged 5 commits intomasterfrom
longer-title-remark-history

Conversation

@PaulWessel
Copy link
Member

@PaulWessel PaulWessel commented Dec 3, 2021

See #6078 for motivation and context. The length limits on the three grid header attributes title, command, and remark were set back in the last century and because they are fixed array sizes in the GMT grid header they cannot be changed without breaking compatibility. The solution is to add pointers to the hidden grid header structure that is not part of the API and hence store strings that exceed the limits here. This PR does this juggling where we may need to use one of the other of the strings.

Tests pass, and longer strings work and give a warning:

gmt grdedit -D+t"I am going to add a very long title that should be much longer than 80 characters, in fact it is probably way longer than 100 by the time I am tired of adding random words"+r"Junk that changes the structures starting at is a breaking change. But the C struct GMT_GRID_HEADER could eventually be used to add/read new attributes in the nc files with extended info" t.nc
grdedit [WARNING]: Title string exceeds upper length of 80 characters (will truncated in non-netCDF grid files)
grdedit [WARNING]: Remark string exceeds upper length of 160 characters (truncated)

gmt grdinfo t.nc
t.nc: Title: I am going to add a very long title that should be much longer than 80 characters, in fact it is probably way longer than 100 by the time I am tired of adding random words
t.nc: Command: grdmath -R0/2/0/2 -I1 X = t.nc
t.nc: Remark: Junk that changes the structures starting at is a breaking change. But the C struct GMT_GRID_HEADER could eventually be used to add/read new attributes in the nc files with extended info
t.nc: Gridline node registration used [Cartesian grid]
t.nc: Grid file format: nf = GMT netCDF format (32-bit float), CF-1.7
t.nc: x_min: 0 x_max: 2 x_inc: 1 name: x n_columns: 3
t.nc: y_min: 0 y_max: 2 y_inc: 1 name: y n_rows: 3
t.nc: v_min: 0 v_max: 2 name: z
t.nc: scale_factor: 1 add_offset: 0
t.nc: format: classic

Converting to native binary, for instance, will only give the truncated versions.

This PR will also need to be testing in GMT.jl (@joa-quim) and PyGMT (@meghanrjones) to make sure there are no surprises.

@PaulWessel
Handle longer strings for netCDF history, remark and title
7af72bd 
The length limits on these three were set back in the last century and because they are fixed array sizes in the GMT grid header they cannot be changed without breaking compatibility.  The solution is to add pointers to the hidden grid header structure that is not part of the API and hence store strings that exceed the limits here.  This PR does this juggling where we may need to use one of the other of the strings.
@PaulWessel
polish
8ba5e7f
@PaulWessel PaulWessel added the enhancement Improving an existing feature label Dec 3, 2021
@PaulWessel PaulWessel added this to the 6.4.0 milestone Dec 3, 2021
@PaulWessel PaulWessel self-assigned this Dec 3, 2021
@maxrjones
Copy link
Member

maxrjones commented Dec 3, 2021

No impact to PyGMT based on existing tests.

@PaulWessel
Copy link
Member Author

PaulWessel commented Dec 4, 2021

Sorry, just added a commit since I had messed up the GMT command history. Now works and consistently will give warnings regardless of where things are initialized or reset. I tested the history limit 320 with a long repetitive grdmath call and it works:

The netCDF grid retains the full command via the hidden struct:

gmt grdmath -R0/20/0/20 -I1 X 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB = t.grd
grdmath [WARNING]: Command string exceeds upper length of 320 characters (will be truncated in non-netCDF grid files)
(base) pwessel@macnut:~-> gmt grdinfo t.grd
t.grd: Title: Produced by grdmath
t.grd: Command: grdmath -R0/20/0/20 -I1 X 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB = t.grd
t.grd: Remark: 
t.grd: Gridline node registration used [Cartesian grid]
t.grd: Grid file format: nf = GMT netCDF format (32-bit float), CF-1.7
t.grd: x_min: 0 x_max: 20 x_inc: 1 name: x n_columns: 21
t.grd: y_min: 0 y_max: 20 y_inc: 1 name: y n_rows: 21
t.grd: v_min: 0 v_max: 20 name: z
t.grd: scale_factor: 1 add_offset: 0
t.grd: format: classic

Use native binary grid which will be truncated:

(base) pwessel@macnut:~-> gmt grdmath -R0/20/0/20 -I1 X 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB = t.grd=bf
grdmath [WARNING]: Command string exceeds upper length of 320 characters (will be truncated in non-netCDF grid files)
(base) pwessel@macnut:~-> gmt grdinfo t.grd=bf
t.grd: Title: Produced by grdmath
t.grd: Command: grdmath -R0/20/0/20 -I1 X 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 1000000 ADD 1000000 SUB 10000
t.grd: Remark: 
t.grd: Gridline node registration used [Cartesian grid]
t.grd: Grid file format: bf = GMT native, C-binary format (32-bit float)
t.grd: x_min: 0 x_max: 20 x_inc: 1 name: x n_columns: 21
t.grd: y_min: 0 y_max: 20 y_inc: 1 name: y n_rows: 21
t.grd: v_min: 0 v_max: 20 name: z
t.grd: scale_factor: 1 add_offset: 0

@joa-quim
Copy link
Member

joa-quim commented Dec 4, 2021

No problem with Julia either.

@PaulWessel PaulWessel merged commit 1e9dd0a into master Dec 4, 2021
@PaulWessel PaulWessel deleted the longer-title-remark-history branch December 4, 2021 01:40
@maxrjones maxrjones added the add-changelog Add PR to the changelog label Jan 25, 2022
@maxrjones maxrjones mentioned this pull request Mar 18, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxrjones maxrjones maxrjones approved these changes

@joa-quim joa-quim Awaiting requested review from joa-quim

@seisman seisman Awaiting requested review from seisman

@Esteban82 Esteban82 Awaiting requested review from Esteban82

Assignees

@PaulWessel PaulWessel

Labels

add-changelog Add PR to the changelog enhancement Improving an existing feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@PaulWessel @maxrjones @joa-quim