Skip to content

Commit 2329c9e

Browse files
jsquaredosquaredjohanneskoester
jsquaredosquared
and
johanneskoester
authored
feat: Support Hy in script directive (snakemake#3824)
<!--Add a description of your PR here--> Add support for [Hy](https://hylang.org/) in the script directive. This will allow the use of a Lisp-like syntax for writing scripts, e.g.: ```hy (require hyrule [-> ->>]) (defn is-odd? [n] (!= (% n 2) 0)) (setv result (->> (get snakemake.input 0) open .readlines (map int) (filter is-odd?) sum)) (print result :file (-> (get snakemake.output 0) (open "w"))) ``` ### QC <!-- Make sure that you can tick the boxes below. --> * [X] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [X] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runtime support for Hy scripts. * **Documentation** * Added Hy and Xonsh language sections with examples and cross-references. * Fixed Bash example formatting and removed a duplicated Xonsh example. * **Tests** * Added end-to-end test exercising Hy script execution with a Snakefile, Hy script, and sample input/output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Johannes Köster <johannes.koester@uni-due.de> Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>
1 parent a72dd0e commit 2329c9e

6 files changed

Lines changed: 159 additions & 33 deletions

File tree

‎docs/snakefiles/rules.rst‎

Lines changed: 84 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1400,6 +1400,88 @@ or using the explicit import:
14001400
14011401
You can use the Python debugger from within the script if you invoke Snakemake with ``--debug``.
14021402

1403+
Xonsh_
1404+
""""""
1405+
1406+
.. _Xonsh: https://xon.sh
1407+
1408+
Because Xonsh is a superset of Python, you can use a Xonsh script as you would a Python script, but with all the additional shell primitives that Xonsh provides.
1409+
1410+
For example, with this rule:
1411+
1412+
.. code-block:: python
1413+
1414+
rule get_variants_in_genes:
1415+
input:
1416+
vcf="input.vcf",
1417+
gene_locations="genes.bed",
1418+
output:
1419+
"output.tsv"
1420+
conda:
1421+
"envs/variant_calling.yaml"
1422+
log:
1423+
"logs/get_variants_in_genes.log"
1424+
script:
1425+
"scripts/get_variants_in_genes.xsh"
1426+
1427+
the Xonsh script might look like this:
1428+
1429+
.. code-block::
1430+
1431+
$XONSH_TRACEBACK_LOGFILE = snakemake.log[0]
1432+
1433+
annotations = ", ".join(
1434+
f'ANN["{field}"]' for field in ["Consequence", "SYMBOL", "Feature", "BIOTYPE"]
1435+
)
1436+
1437+
bcftools view -R @(snakemake.input.gene_locations) @(snakemake.input.vcf) \
1438+
| vembrane table --output @(snakemake.output[0]) @(f'CHROM, POS, ID, {annotations}')
1439+
1440+
1441+
Hy_
1442+
"""
1443+
1444+
.. _Hy: https://hylang.org/
1445+
1446+
Hy allows you to interact with Python using a Lisp-like syntax.
1447+
1448+
For example, with this rule:
1449+
1450+
.. code-block:: python
1451+
1452+
rule get_sum_of_odd_numbers:
1453+
input:
1454+
"list_of_numbers.txt"
1455+
output:
1456+
results_file="sum_of_odd_numbers.txt"
1457+
conda:
1458+
"envs/hy.yaml"
1459+
script:
1460+
"scripts/sum_odd_numbers.hy"
1461+
1462+
the Hy script might look like this:
1463+
1464+
.. code-block:: hy
1465+
1466+
(require hyrule [-> ->>])
1467+
1468+
(defn is-odd? [n] (!= (% n 2) 0))
1469+
1470+
(setv result
1471+
(->> (get snakemake.input 0)
1472+
open
1473+
.readlines
1474+
(map int)
1475+
(filter is-odd?)
1476+
sum))
1477+
1478+
(with [f
1479+
(-> (get snakemake.output "results_file")
1480+
(open "w"))]
1481+
(print result :file f))
1482+
1483+
1484+
14031485
R and R Markdown
14041486
~~~~~~~~~~~~~~~~
14051487

@@ -1654,7 +1736,7 @@ remaining directives can be found in the variable ``snakemake``.
16541736
As arrays cannot be nested in Bash, use of python's ``dict`` in directives is not supported. So, adding a ``params`` key of ``data={"foo": "bar"}`` will not be reflected - ``${snakemake_params[data]}`` actually only returns ``"foo"``.
16551737

16561738
Bash Example 1
1657-
~~~~~~~~~~~~~~
1739+
""""""""""""""
16581740

16591741
.. code-block:: python
16601742
@@ -1697,7 +1779,7 @@ double-quote any variable that could contain a file name. However, `in some case
16971779
such as ``${snakemake_params[opts]}`` in the above example.
16981780

16991781
Bash Example 2
1700-
~~~~~~~~~~~~~~
1782+
""""""""""""""
17011783

17021784
.. code-block:: python
17031785
@@ -1752,26 +1834,6 @@ as ``"${snakemake_input[0]}"`` and ``"${snakemake_input[1]}"``.
17521834
For technical reasons, scripts are executed in ``.snakemake/scripts``. The original script directory is available as ``scriptdir`` in the ``snakemake`` object.
17531835

17541836

1755-
Xonsh_
1756-
~~~~~~
1757-
1758-
.. _Xonsh: https://xon.sh
1759-
1760-
.. code-block:: python
1761-
1762-
rule NAME:
1763-
input:
1764-
"path/to/inputfile",
1765-
"path/to/other/inputfile"
1766-
output:
1767-
"path/to/outputfile",
1768-
"path/to/another/outputfile"
1769-
script:
1770-
"path/to/script.xsh"
1771-
1772-
Because Xonsh is a superset of Python, you can use a Xonsh script as you would a Python script (see above), but with all the additional shell primitives that Xonsh provides.
1773-
1774-
17751837
.. _snakefiles_notebook-integration:
17761838

17771839
Jupyter notebook integration

‎src/snakemake/script/__init__.py‎

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,30 +5,32 @@
55

66
import collections
77
import itertools
8-
import math
9-
import os
10-
from numbers import Integral, Real, Complex, Number
11-
from collections.abc import Iterable
128
import json
9+
import math
1310
import os
1411
import pickle
1512
import re
13+
import shlex
1614
import sys
1715
import tempfile
1816
import textwrap
1917
import typing
20-
import shlex
18+
import urllib.parse
2119
from abc import ABC, abstractmethod
2220
from collections.abc import Iterable
21+
from numbers import Complex, Integral, Number, Real
2322
from pathlib import Path
24-
from typing import List, Optional, Pattern, Tuple, Union, Dict
23+
from typing import Dict, List, Optional, Pattern, Tuple, TypeVar, Union
2524
from urllib.error import URLError
26-
import urllib.parse
27-
from typing import TypeVar
2825

2926
from snakemake import io as io_
3027
from snakemake import sourcecache
31-
from snakemake.common import MIN_PY_VERSION, ON_WINDOWS, get_snakemake_searchpaths
28+
from snakemake.common import (
29+
MIN_PY_VERSION,
30+
ON_WINDOWS,
31+
get_report_id,
32+
get_snakemake_searchpaths,
33+
)
3234
from snakemake.deployment import singularity
3335
from snakemake.exceptions import WorkflowError
3436
from snakemake.logging import logger
@@ -40,7 +42,6 @@
4042
infer_source_file,
4143
)
4244
from snakemake.utils import format
43-
from snakemake.common import get_report_id
4445

4546
# TODO use this to find the right place for inserting the preamble
4647
PY_PREAMBLE_RE = re.compile(r"from( )+__future__( )+import.*?(?P<end>[;\n])")
@@ -1616,6 +1617,15 @@ def execute_script(self, fname, edit=False):
16161617
)
16171618

16181619

1620+
class HyScript(PythonScript):
1621+
def write_script(self, preamble, fd):
1622+
fd.write(f"(pys #[[{preamble}]])".encode())
1623+
fd.write(self.source.encode())
1624+
1625+
def execute_script(self, fname, edit=False):
1626+
self._execute_cmd("hy {fname:q}", fname=fname)
1627+
1628+
16191629
def strip_re(regex: Pattern, s: str) -> Tuple[str, str]:
16201630
"""Strip a substring matching a regex from a string and return the stripped part
16211631
and the remainder of the original string.
@@ -1680,6 +1690,8 @@ def get_language(source_file, source):
16801690
language = "bash"
16811691
elif filename.endswith(".xsh"):
16821692
language = "xonsh"
1693+
elif filename.endswith(".hy"):
1694+
language = "hy"
16831695

16841696
# detect kernel language for Jupyter Notebooks
16851697
if language == "jupyter":
@@ -1744,10 +1756,11 @@ def script(
17441756
"rust": RustScript,
17451757
"bash": BashScript,
17461758
"xonsh": XonshScript,
1759+
"hy": HyScript,
17471760
}.get(language, None)
17481761
if exec_class is None:
17491762
raise ValueError(
1750-
"Unsupported script: Expecting either Python (.py), R (.R), RMarkdown (.Rmd), Julia (.jl), Rust (.rs), Bash (.sh), or Xonsh (.xsh) script."
1763+
"Script must be one of the following filetypes: [.py .R .Rmd .jl .rs .sh .xsh .hy]"
17511764
)
17521765

17531766
executor = exec_class(

‎tests/test_script_hy/Snakefile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
rule all:
2+
input:
3+
"test.out"
4+
5+
6+
rule test_hy:
7+
input:
8+
"test.in"
9+
output:
10+
"test.out"
11+
conda:
12+
"envs/hy.yaml"
13+
script:
14+
"scripts/test.hy"

‎tests/test_script_hy/expected-results/test.out‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
100

‎tests/test_script_hy/scripts/test.hy‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
(require hyrule [-> ->>])
2+
3+
(defn is-odd? [n] (!= (% n 2) 0))
4+
5+
(setv result
6+
(->> (get snakemake.input 0)
7+
open
8+
.readlines
9+
(map int)
10+
(filter is-odd?)
11+
sum))
12+
13+
(print result
14+
:file
15+
(-> (get snakemake.output 0)
16+
(open "w")))

‎tests/test_script_hy/test.in‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
1
2+
2
3+
3
4+
4
5+
5
6+
6
7+
7
8+
8
9+
9
10+
10
11+
11
12+
12
13+
13
14+
14
15+
15
16+
16
17+
17
18+
18
19+
19
20+
20

0 commit comments

Comments
 (0)