Skip to content

updated velvet wrappers#503

Closed
bernt-matthias wants to merge 51 commits intogalaxyproject:masterfrom
bernt-matthias:topic/velvet-impro
Closed

updated velvet wrappers#503
bernt-matthias wants to merge 51 commits intogalaxyproject:masterfrom
bernt-matthias:topic/velvet-impro

Conversation

@bernt-matthias
Copy link
Contributor

Some first steps toward an up-to-date velvet wrapper

  • use GALAXY_SLOTS
  • removed python wrapper script
  • allow input of separate paired end reads
  • allow larger k-mer size up to the limit of the conda requirement (note: even inputs are handled by velvet by subtracting 1 -- should be fine)

Open:

I'm unsure how the output directory is handled. Seemed to work before. I just do not understand it. Could someone point me to the documentation?

Any other ideas. I will take care of velvetg later.

davebx and others added 2 commits July 10, 2017 15:37
- use GALAXY_SLOTS
- removed python wrapper script
- allow input of separate paired end reads
Copy link
Member

@bgruening bgruening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bernt-matthias do you have time to add some tests?
This would be super awesome!

Also we should move this tool into IUC - but we can do this after merging this PR if you like.

@bernt-matthias
Copy link
Contributor Author

I would definitely like to add a test. Since there is already a test (but the data is missing) I was wondering if the data might still be somewhere. Any ideas? Otherwise I would take these: http://galaxyproject.github.io/training-material/topics/assembly/tutorials/general-introduction/tutorial.html

@bernt-matthias
Copy link
Contributor Author

There is the possibility to input raw (and raw.gz) data. Any idea if there is a Galaxy data type for this?

@bgruening
Copy link
Member

No idea about the test data, sorry.
raw is very inexplicit do you know how this looks like?

seems so. at least velvet outputs this on startup. was wondering if we can parse this and use the value in the form. is there such a possibility?

In theory yes, in practice this function is deprecated :)

Looping in @Slugger70 here, he has used this tool a lot in the past.

@bernt-matthias
Copy link
Contributor Author

For raw: I can only guess from a peek into the velvet sources that it is sequence data without any header info. I guess this is used only rareley. Also for raw no separate input of paired data seems possible. I guess we leave it out..?

@nsoranzo
Copy link
Member

@Slugger70
Copy link

I'd leave Raw out. I've never seen it used.. Bioconda Velvet is compiled with MAXKMERLENGTH=191, CATEGORIES=4, OPENMP=1, LONGSEQUENCES=1. You use the OPENMP settings by setting a couple of env vars. See the VelvetOptimiser wrapper for details.

        export OMP_NUM_THREADS=2 &&
        export OMP_THREAD_LIMIT=2 &&
        VelvetOptimiser.pl
            -t "\${GALAXY_SLOTS:-1}"
            -s $start_kmer
            -e $end_kmer
            -d out
            -f "

I couldn't figure out how to make the omp_num_threads set in a more dynamic way..

- improved stdio (including oom check)
- added ins_length3|4[_sd]
- and further tests for velveth
@bernt-matthias
Copy link
Contributor Author

bernt-matthias commented Feb 20, 2018

The velveth wrapper should now be functional.

  • threading is enabled with export OMP_NUM_THREADS="\${GALAXY_SLOTS:-1}" &&. Unfortunately velvet uses one more thread than specified this way, but setting OMP_THREAD_LIMIT the same way seems to be dysfunctional in if the value is 1. Can we compute here? Then we could +1?
  • I took a down sampled version of the GTN test data

Some questions on velvetg

  • There was <option value="yes" selected="${input.metadata.paired_end_reads}">Yes</option> Does such a thing work? Linting fails with this.
  • The test currently fails because the ftype="velvet" seems not to work. The data type seems odd to me: it consists of a (quite useless) html file and the Sequences and Roadmaps file in a sub directory. The velveth wrapper currently relies on the fact that the html file is seemingly autogenerated by the velvet data type class? (In the original version of this wrapper the velveth_wrapper.py script did this)
  • I guess some of the conditions are unnecessary and can be replaced by simple bools?
  • there is a TODO with some parameters which are currently not implemented in the wrapper. Should we include them? I guess they are related to the columbus extensions mentioned above.

@bgruening
Copy link
Member

@bernt-matthias I tried to enable testing for this repo.

@nsoranzo
Copy link
Member

@lparsons
Copy link
Contributor

lparsons commented Mar 1, 2018

Could/should this be moved to https://github.com/galaxyproject/tools-iuc?

@bernt-matthias
Copy link
Contributor Author

Could: Yes. Should: I don't know.

Any comments on the question in my last comment? Then I could finish the work in this PR.

@bgruening
Copy link
Member

Moving to IUC: yes. But I think we can also do this after this PR.
I can not answer your questions, but @Slugger70 probably can. I'm wondering if velvet is worth to update nowadays.

@bernt-matthias
Copy link
Contributor Author

I think its worth -- and if its only for the GTN tutorials.

peterjc and others added 23 commits October 15, 2018 12:58
Rename .tt_blacklist to .tt_skip (tools we don't test on Travis)
to workaround `planemo test --update_test_data` inability to update
the files in the subdirectories.
* column_maker allow header

* column_maker allow header - fix sed expr

* column_maker allow header - allow to rename the new column

* fix quote
- use GALAXY_SLOTS
- removed python wrapper script
- allow input of separate paired end reads
- improved stdio (including oom check)
- added ins_length3|4[_sd]
- and further tests for velveth
- velvetg needs copy of input, otherwise it writes to input
- added missing test data
bernt-matthias added a commit to bernt-matthias/tools-iuc that referenced this pull request Jun 9, 2020
bernt-matthias added a commit to bernt-matthias/tools-iuc that referenced this pull request Jun 10, 2020
bernt-matthias added a commit to bernt-matthias/tools-devteam that referenced this pull request Jun 10, 2020
bernt-matthias added a commit to bernt-matthias/tools-devteam that referenced this pull request Jun 10, 2020
@bernt-matthias
Copy link
Contributor Author

closed via #564

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.