Skip to content

Let batch -D drop the mandatory moving of product files#6933

Merged
PaulWessel merged 3 commits intomasterfrom
prevent-move
Aug 3, 2022
Merged

Let batch -D drop the mandatory moving of product files#6933
PaulWessel merged 3 commits intomasterfrom
prevent-move

Conversation

@PaulWessel
Copy link
Member

See #6932 for discussion. This PR implements a -D option that stops any copying of batch products to the top directory. This solution does not preempt further consideration of the other suggestion which is to generalize the naming of product files via a -Ftemplate option.

See #6932 for discussion.  This PR implements a -D option that stops any copying.  This solution does not preempt further consideration of the other suggestion which is to generalize the naming of product files via a -Ftemplate option.
@PaulWessel PaulWessel added the new core module feature PR that implements a new core module feature label Aug 3, 2022
@PaulWessel PaulWessel added this to the 6.5.0 milestone Aug 3, 2022
@PaulWessel PaulWessel requested a review from Esteban82 August 3, 2022 13:48
@PaulWessel PaulWessel self-assigned this Aug 3, 2022
@Esteban82
Copy link
Member

It would be possible that -D should be used with -Q?

If I don't use -Q, I think that all files are deleted. It could be possible?

@PaulWessel
Copy link
Member Author

PaulWessel commented Aug 3, 2022

I have also implemented -F and testing it. The -Q should strictly be for debugging. With -D (this PR) and -F (next PR coming once this is approved) we have full control:

  1. If you are happy with producing single named products like job_00012.txt and job_0012.grd etc, then there are no changes. These products are moved up to the top dir when the jobs complete.
  2. If your script is not relying on this naming and is doing something custom (like mine) then you use -D to avoid batch trying to consolidate files that are not there. batch cannot know if there should be files or not so this takes an option.
  3. If your script wishes to name the products using the input -Ttimefile then you use -Ftemplate to do so. That adds more flexibility for batch jobs that do not just depend on a single parameter. -F can be combined with -D if you are writing the files to some custom place yourself.

With these changes I can write batch jobs that produces parameter files like this:

# Parameter file for job 2109
BATCH_DIR=EFLX3_2109
BATCH_NAME=E_obl_flex_G1.04_I2660_10.0k
BATCH_JOB=2109
BATCH_ITEM=2109
BATCH_COL0=1.04
BATCH_COL1=2660
BATCH_COL2=10

You will note I have introduced a new parameter called BATCH_DIR which basically replaces the old use of BATCH_NAME and BATCH_NAME is now exclusively used for product naming.

@PaulWessel PaulWessel merged commit ec0cd66 into master Aug 3, 2022
@PaulWessel PaulWessel deleted the prevent-move branch August 3, 2022 15:00
@maxrjones maxrjones added the add-changelog Add PR to the changelog label Dec 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

add-changelog Add PR to the changelog new core module feature PR that implements a new core module feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants