Allow API import/export of format 2 workflows.#6776
Merged
mvdbeek merged 8 commits intogalaxyproject:devfrom Oct 26, 2018
Merged
Allow API import/export of format 2 workflows.#6776mvdbeek merged 8 commits intogalaxyproject:devfrom
mvdbeek merged 8 commits intogalaxyproject:devfrom
Conversation
a90464c to
bc9717b
Compare
50bdfd5 to
eb0ddac
Compare
540706a to
3be0747
Compare
3be0747 to
397f237
Compare
397f237 to
2fac59a
Compare
Avoid re-arranging dictionaries when JSON is being handled for instance.
2fac59a to
f7415cb
Compare
This was referenced Oct 22, 2018
mvdbeek
approved these changes
Oct 26, 2018
Member
Author
|
Thanks a million @mvdbeek ! |
nsoranzo
added a commit
to nsoranzo/galaxy
that referenced
this pull request
Nov 30, 2018
with `make config-rebuild` . Follow-up on galaxyproject#6776 .
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on a set of Galaxy PRs #6746, #6807, #6811 and a series of gxformat2 releases.
In addition to bringing in the latest changes to gxformat2 to allow these syntax changes (described below) and updating hundreds of lines of test workflows - this PR finally enables such workflows to work directly with the Galaxy API (switched on
enable_beta_workflow_format). When enabled, workflows imported from dictionaries will now be checked and if they look like a format2 workflow they will be pre-converted to a native format before import - uploading YAML wrapped in JSON to avoid dictionary handling changes is allowed also. Likewise, there is a couple new styles of download that attempt to extract format 2 workflows from the native representation (using new functionality added to recent versions of gxformat2). These can be downloaded as JSON or as YAML string content wrapped up in JSON in order to preserve the pretty YAML formatting and dictionary ordering implemented in gxformat2.If the JSON wrapped YAML-in-strings seems odd - consider also an important future direction of this work is likely to store the original supplied YAML alongside the native representation of the workflow and then allow the workflow editor to operate a series of deltas to both in parallel. This can be done with a round trip aware YAML parsing/writer such as ruamel.yaml so that user comments and formatting as well as extraneous data in the YAML are preserved.
This PR continues to refine the workflow syntax toward a more concise and CWL compatible syntax. The following two code blocks are a before and after example demonstrating the syntax changes.
Before this series of PRs:
My recommended best practice after this PR is merged:
Prior to this pull request, all subworkflows (format-version 0.1 or 2) would be imported repeatedly - once per step that referenced them. This PR introduces a CWL-derived syntax for repeatedly referencing the same workflow and updates the Galaxy import functionality to properly resolve these references and import such workflows only once instead of once per step.