Skip to content

fix: get rid of CRLF#5494

Merged
ikatyang merged 29 commits intoprettier:masterfrom
ikatyang:fix/crlf
Dec 8, 2018
Merged

fix: get rid of CRLF#5494
ikatyang merged 29 commits intoprettier:masterfrom
ikatyang:fix/crlf

Conversation

@ikatyang
Copy link
Copy Markdown
Member

@ikatyang ikatyang commented Nov 16, 2018

  • get rid of CRLF parsing/printing
    • normalize end of line before parsing, offset-related fields (e.g., rangeStart, cursorOffset, etc.) are recalculated if there're CRs.
    • replace string doc \n with options.endOfLine before printDocToString if options.endOfLine is not lf
  • formatRange: end of lines are normalized now even if it's not in the range. This is the side effect that we need to normalize eol before parsing but formatRange needs AST to calculate the range:

    prettier/src/main/core.js

    Lines 173 to 181 in 6ee2f46

    function formatRange(text, opts) {
    const parsed = parser.parse(text, opts);
    const ast = parsed.ast;
    text = parsed.text;
    const range = rangeUtil.calculateRange(text, opts, ast);
    const rangeStart = range.rangeStart;
    const rangeEnd = range.rangeEnd;
    const rangeString = text.slice(rangeStart, rangeEnd);
  • update yaml parsers (closes chore: update yaml parser #5491)
  • add an env var TEST_CRLF to test CRLF by replacing LF with CRLF before formatting (enabled on Windows CI)
  • normalize end of line in this repo (changes in tests/)

  • I’ve added tests to confirm my change works.
  • (If changing the API or CLI) I’ve documented the changes I’ve made (in the docs/ directory)
  • I’ve read the contributing guidelines.

Try the playground for this PR

@ikatyang ikatyang added the status:wip Pull requests that are a work in progress and should not be merged label Nov 16, 2018
@ikatyang ikatyang mentioned this pull request Nov 16, 2018
2 tasks
@kachkaev
Copy link
Copy Markdown
Member

Is the plan to visualize end of line in every snapshot? Shall most of the test files be turned to LF for consistency?

It’d be great if Prettier repo could follow endOfLine: lf itself with few exceptions when CRLF testing is needed. This cleanup comes at a cost of breaking git blame though.

@ikatyang
Copy link
Copy Markdown
Member Author

Some initial thoughts:

  • Replace all CRLF with LF in this repo
  • Set * text=auto in .gitattributes. If I understand correctly, it allows us to have consistent LF on Unix and consistent CRLF on Windows.

    (https://stackoverflow.com/a/38017715)
    text=auto: when someone commits a file, Git guesses whether that file is a text file or not, and if it is, it will commit a version of the file where all CR + LF bytes are replaced with LF bytes. It doesn't directly affect what files look like in the working tree, there are other settings that will convert LF bytes to CR + LF bytes when checking out a file.

  • Visualize whitespaces for comparison (i.e., expect(visualize(a)).toEqual(visualize(b))), we could observe their diffs if the test failed.
  • Jest normalizes line endings for snapshots so we don't need to handle it. (no visualization)

@j-f1
Copy link
Copy Markdown
Member

j-f1 commented Nov 16, 2018

Seems like Azure Pipelines didn’t run for some reason.

@kachkaev
Copy link
Copy Markdown
Member

kachkaev commented Nov 16, 2018

Jest normalizes line endings for snapshots so we don't need to handle it.

Jest simply kills all CRs in snapshots – see its normalizeNewlines() function. We can't really check CR/CRLF with this, and this is why replace(/\r/g, "/*CR*/") was necessary. Perhaps, we could use visualizeEndOfLine() in snapshots too?

Set * text=auto in .gitattributes.

I believe it should be * text=auto eol=lf, followed by a couple of specific rules with eol=crlf / eol=crlf for some CRLF test files. Alternatively, these specific .gitattributes can go directly to folders with particular tests. And yes, in this case all source files and the majority of test files will need to be converted from CRLF to LF at a cost of 'losing' git blame on master.

Ideally, all files on all operating systems should be checked-out with identical line endings, regardless of git config --global core.autocrlf XXX. This can be checked in CI too.

@ikatyang
Copy link
Copy Markdown
Member Author

@j-f1 Its message looks confusing but it actually works, you need to click it to see the detail logs.


@kachkaev

Jest simply kills all CRs in snapshots – see its normalizeNewlines() function. We can't really check CR/CRLF with this, and this is why replace(/\r/g, "/*CR*/") was necessary. Perhaps, we could use visualizeEndOfLine() in snapshots too?

I meant, the CRLF/LF checks are done in expect(visualize(a)).toEqual(visualize(b)), snapshots are used for visible records. And that's also the reason why I don't want to use eol=lf, I'd like to use CRLF on our Windows CI to test CRLF.

@ikatyang
Copy link
Copy Markdown
Member Author

Or maybe we should use eol=lf and add another test to replace LF with CRLF before formatting?

@kachkaev
Copy link
Copy Markdown
Member

IMO we can use eol=lf in the whole repo except a couple of test files where we're checking endOfLine option.

- template: .azure-pipelines/dev.yml

# There're issues for unicode on Windows build
# https://github.com/Microsoft/azure-pipelines-tasks/issues/8534
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows:

image

Linux:

image

@ikatyang
Copy link
Copy Markdown
Member Author

Based on #5498 (comment), it seems we should revert 38bfa05, but I'm not sure how to deal with those bad unicode. Should we use <CRLF> instead of ?

@kachkaev
Copy link
Copy Markdown
Member

copied a couple of comments from #5498 as asked by @ikatyang

We shouldn't need to configure global settings and we should be able to test CRLF locally even if it's on LF system (via TEST_CRLF).

Testing on Windows machines with globally configured CRLF and LF in git is what I meant. This can be useful because Windows folks have different preferencess and all of them should be able to contribute to Prettier without problems of running local tests. By default, git’s core.autocrlf is true, but some devs set it to false, e.g. create-react-app contribution guidelines recommend doing so.

I did not know Azure pipelines had a flag for CRLF, interesting!


Running tests on a Windows machine with LF ia different than doing this on Linux, because on Windows os.EOL is CRLF regardless of how git is configured. Some modules refer to this value and this can affect what Prettier generates in the end.

@ikatyang ikatyang changed the title [WIP] fix: CRLF fix: get rid of CRLF Nov 21, 2018
@ikatyang ikatyang removed the status:wip Pull requests that are a work in progress and should not be merged label Nov 21, 2018
@ikatyang
Copy link
Copy Markdown
Member Author

Updated the OP to reflect the changes, ready for review.
(Most of the diffs are line ending normalization, just ignore them.)

@ikatyang ikatyang added this to the 1.16 milestone Nov 29, 2018
@ikatyang ikatyang merged commit 952bc0c into prettier:master Dec 8, 2018
@ikatyang ikatyang deleted the fix/crlf branch December 8, 2018 10:28
@j-f1 j-f1 mentioned this pull request Jan 15, 2019
@lock lock bot added the locked-due-to-inactivity Please open a new issue and fill out the template instead of commenting. label Mar 8, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Mar 8, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

locked-due-to-inactivity Please open a new issue and fill out the template instead of commenting.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants