Merged
Conversation
added 2 commits
April 1, 2023 10:54
MarcoGorelli
commented
Apr 1, 2023
MarcoGorelli
commented
Apr 1, 2023
MarcoGorelli
commented
Apr 1, 2023
MarcoGorelli
commented
Apr 1, 2023
added 2 commits
April 1, 2023 13:37
MarcoGorelli
commented
Apr 1, 2023
added 2 commits
April 1, 2023 18:45
32a872f to
869f897
Compare
added 2 commits
April 2, 2023 10:42
e055841 to
b680d76
Compare
MarcoGorelli
commented
Apr 2, 2023
nbqa/__main__.py
Outdated
Comment on lines
+598
to
+607
| with open(file_name) as fd: | ||
| for line in fd: | ||
| if line == CODE_SEPARATOR: | ||
| parsed_cells.append(current_cell) | ||
| current_cell = [] | ||
| current_cell.append(line) | ||
| parsed_cells.append("".join(current_cell)) | ||
| if parsed_cells: | ||
| # skip the initial empty one | ||
| parsed_cells = parsed_cells[1:] |
Collaborator
Author
There was a problem hiding this comment.
way too complicated, just return result from first pass?
2f0754a to
4c98087
Compare
2e91a4c to
3ee1b66
Compare
Collaborator
Author
|
seems to be fine, from doing some integration tests |
Collaborator
Author
|
seems fine, let's ship it and see what happens |
6 tasks
dhruvmanila
added a commit
to astral-sh/ruff
that referenced
this pull request
Jun 12, 2023
## Summary Add support for applying auto-fixes in Jupyter Notebook. ### Solution Cell offsets are the boundaries for each cell in the concatenated source code. They are represented using `TextSize`. It includes the start and end offset as well, thus creating a range for each cell. These offsets are updated using the `SourceMap` markers. ### SourceMap `SourceMap` contains markers constructed from each edits which tracks the original source code position to the transformed positions. The following drawing might make it clear:  The center column where the dotted lines are present are the markers included in the `SourceMap`. The `Notebook` looks at these markers and updates the cell offsets after each linter loop. If you notice closely, the destination takes into account all of the markers before it. The index is constructed only when required as it's only used to render the diagnostics. So, a `OnceCell` is used for this purpose. The cell offsets, cell content and the index will be updated after each iteration of linting in the mentioned order. The order is important here as the content is updated as per the new offsets and index is updated as per the new content. ## Limitations ### 1 Styling rules such as the ones in `pycodestyle` will not be applicable everywhere in Jupyter notebook, especially at the cell boundaries. Let's take an example where a rule suggests to have 2 blank lines before a function and the cells contains the following code: ```python import something # --- def first(): pass def second(): pass ``` (Again, the comment is only to visualize cell boundaries.) In the concatenated source code, the 2 blank lines will be added but it shouldn't actually be added when we look in terms of Jupyter notebook. It's as if the function `first` is at the start of a file. `nbqa` solves this by recording newlines before and after running `autopep8`, then running the tool and restoring the newlines at the end (refer nbQA-dev/nbQA#807). ## Test Plan Three commands were run in order with common flags (`--select=ALL --no-cache --isolated`) to isolate which stage the problem is occurring: 1. Only diagnostics 2. Fix with diff (`--fix --diff`) 3. Fix (`--fix`) ### https://github.com/facebookresearch/segment-anything ``` ------------------------------------------------------------------------------- Jupyter Notebooks 3 0 0 0 0 |- Markdown 3 98 0 94 4 |- Python 3 513 468 4 41 (Total) 611 468 98 45 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/segment-anything/**/*.ipynb --fix ... Found 180 errors (89 fixed, 91 remaining). ``` ### https://github.com/openai/openai-cookbook ``` ------------------------------------------------------------------------------- Jupyter Notebooks 65 0 0 0 0 |- Markdown 64 3475 12 2507 956 |- Python 65 9700 7362 1101 1237 (Total) 13175 7374 3608 2193 =============================================================================== ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/openai-cookbook/**/*.ipynb --fix error: Failed to parse /path/to/openai-cookbook/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb:cell 4:29:18: unexpected token '-' ... Found 4227 errors (2165 fixed, 2062 remaining). ``` ### https://github.com/tensorflow/docs ``` ------------------------------------------------------------------------------- Jupyter Notebooks 150 0 0 0 0 |- Markdown 1 55 0 46 9 |- Python 1 402 289 60 53 (Total) 457 289 106 62 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-docs/**/*.ipynb --fix error: Failed to parse /path/to/tensorflow-docs/site/en/guide/extension_type.ipynb:cell 80:1:1: unexpected token Indent error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/eager/custom_layers.ipynb:cell 20:1:1: unexpected token Indent error: Failed to parse /path/to/tensorflow-docs/site/en/guide/data.ipynb:cell 175:5:14: unindent does not match any outer indentation level error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/representation/unicode.ipynb:cell 30:1:1: unexpected token Indent ... Found 12726 errors (5140 fixed, 7586 remaining). ``` ### https://github.com/tensorflow/models ``` ------------------------------------------------------------------------------- Jupyter Notebooks 46 0 0 0 0 |- Markdown 1 11 0 6 5 |- Python 1 328 249 19 60 (Total) 339 249 25 65 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-models/**/*.ipynb --fix ... Found 4856 errors (2690 fixed, 2166 remaining). ``` resolves: #1218 fixes: #4556
konstin
pushed a commit
to astral-sh/ruff
that referenced
this pull request
Jun 13, 2023
## Summary Add support for applying auto-fixes in Jupyter Notebook. ### Solution Cell offsets are the boundaries for each cell in the concatenated source code. They are represented using `TextSize`. It includes the start and end offset as well, thus creating a range for each cell. These offsets are updated using the `SourceMap` markers. ### SourceMap `SourceMap` contains markers constructed from each edits which tracks the original source code position to the transformed positions. The following drawing might make it clear:  The center column where the dotted lines are present are the markers included in the `SourceMap`. The `Notebook` looks at these markers and updates the cell offsets after each linter loop. If you notice closely, the destination takes into account all of the markers before it. The index is constructed only when required as it's only used to render the diagnostics. So, a `OnceCell` is used for this purpose. The cell offsets, cell content and the index will be updated after each iteration of linting in the mentioned order. The order is important here as the content is updated as per the new offsets and index is updated as per the new content. ## Limitations ### 1 Styling rules such as the ones in `pycodestyle` will not be applicable everywhere in Jupyter notebook, especially at the cell boundaries. Let's take an example where a rule suggests to have 2 blank lines before a function and the cells contains the following code: ```python import something # --- def first(): pass def second(): pass ``` (Again, the comment is only to visualize cell boundaries.) In the concatenated source code, the 2 blank lines will be added but it shouldn't actually be added when we look in terms of Jupyter notebook. It's as if the function `first` is at the start of a file. `nbqa` solves this by recording newlines before and after running `autopep8`, then running the tool and restoring the newlines at the end (refer nbQA-dev/nbQA#807). ## Test Plan Three commands were run in order with common flags (`--select=ALL --no-cache --isolated`) to isolate which stage the problem is occurring: 1. Only diagnostics 2. Fix with diff (`--fix --diff`) 3. Fix (`--fix`) ### https://github.com/facebookresearch/segment-anything ``` ------------------------------------------------------------------------------- Jupyter Notebooks 3 0 0 0 0 |- Markdown 3 98 0 94 4 |- Python 3 513 468 4 41 (Total) 611 468 98 45 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/segment-anything/**/*.ipynb --fix ... Found 180 errors (89 fixed, 91 remaining). ``` ### https://github.com/openai/openai-cookbook ``` ------------------------------------------------------------------------------- Jupyter Notebooks 65 0 0 0 0 |- Markdown 64 3475 12 2507 956 |- Python 65 9700 7362 1101 1237 (Total) 13175 7374 3608 2193 =============================================================================== ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/openai-cookbook/**/*.ipynb --fix error: Failed to parse /path/to/openai-cookbook/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb:cell 4:29:18: unexpected token '-' ... Found 4227 errors (2165 fixed, 2062 remaining). ``` ### https://github.com/tensorflow/docs ``` ------------------------------------------------------------------------------- Jupyter Notebooks 150 0 0 0 0 |- Markdown 1 55 0 46 9 |- Python 1 402 289 60 53 (Total) 457 289 106 62 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-docs/**/*.ipynb --fix error: Failed to parse /path/to/tensorflow-docs/site/en/guide/extension_type.ipynb:cell 80:1:1: unexpected token Indent error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/eager/custom_layers.ipynb:cell 20:1:1: unexpected token Indent error: Failed to parse /path/to/tensorflow-docs/site/en/guide/data.ipynb:cell 175:5:14: unindent does not match any outer indentation level error: Failed to parse /path/to/tensorflow-docs/site/en/r1/tutorials/representation/unicode.ipynb:cell 30:1:1: unexpected token Indent ... Found 12726 errors (5140 fixed, 7586 remaining). ``` ### https://github.com/tensorflow/models ``` ------------------------------------------------------------------------------- Jupyter Notebooks 46 0 0 0 0 |- Markdown 1 11 0 6 5 |- Python 1 328 249 19 60 (Total) 339 249 25 65 ------------------------------------------------------------------------------- ``` ```console $ cargo run --all-features --bin ruff -- check --no-cache --isolated --select=ALL /path/to/tensorflow-models/**/*.ipynb --fix ... Found 4856 errors (2690 fixed, 2166 remaining). ``` resolves: #1218 fixes: #4556
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #806