Skip to content

Jupyter notebook integration leaves empty notebook files when notebook errors occur #348

@dciarletta

Description

@dciarletta

Snakemake version

$ snakemake --version
5.15.0

Describe the bug
This could be viewed as either a bug or a feature request depending on how the jupyter notebook rule was intended to work.

This relates to the usage of the Jupyter notebook integration rules. Currently, if an exception occurs in one of the cells of the template notebook that is being executed and a log: of the notebook is requested, when the rule fails all output is removed, but an empty notebook file is left behind. This makes it difficult to diagnose the problem in the notebook. It would be good to have a copy of the executed notebook left behind with the exception output for diagnosing the problem. This could be achieved if the --allow-errors flag is passed to the call to jupyter-nbconvert at

cmd_tmp = "jupyter-nbconvert --execute {output_parameter} --to notebook --ExecutePreprocessor.timeout=-1 {{fname:q}}".format(

However, adding this flag will also mean that the rule will not fail if an exception is raised in the notebook as long as all outputs are generated. Therefore, it may be desirable to have this flag be added only when specified by the rule declaration.

Logs
n/a

Minimal example
Snakefile:

rule test_notebook_error:
    input:
    output:
        "output"
    log:
        notebook = "generated.ipynb"
    notebook:
        "template.ipynb"

template.ipynb:

{
 "nbformat": 4,
 "nbformat_minor": 2,
 "metadata": {
  "language_info": {
   "name": "python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   }
  },
  "orig_nbformat": 2,
  "file_extension": ".py",
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raise Exception('something went wrong')\n",
    "with open(snakemake.output[0], 'w') as output:\n",
    "    output.write('completed')"
   ]
  }
 ]
}

Running snakemake --cores 1 in a directory with the two files above demonstrates that the rule fails but an empty generated.ipynb is left behind.

Additional context
I'd be happy to submit a pull request to address this issue with some guidance on the preferred approach to allowing errors to occur in the notebook (by default vs explicit request in rule).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions