Skip to content

Update .gitattributes to prefer LF#12816

Merged
seanbudd merged 3 commits into
masterfrom
normalize-line-endings
Jun 27, 2024
Merged

Update .gitattributes to prefer LF#12816
seanbudd merged 3 commits into
masterfrom
normalize-line-endings

Conversation

@seanbudd

@seanbudd seanbudd commented Sep 8, 2021

Copy link
Copy Markdown
Member

Link to issue number:

Part of #12387

Summary of the issue:

The NVDA code base uses a mix of CRLF and LF style line endings, with the preference being CRLF.
LF style endings have been introduced prior to lint checking, and can causes issues when moving code or making changes.
Many IDEs will normalize the line endings when editing files, causing larger than intended diffs of files with mixed line endings.
Instead, we should normalize all line endings of text files to LF and ensure that all files are committed with LF line endings.

Specific files should be excluded from normalization such as:

  • files used by translators, as we are uncertain of the side-effects here
  • binary files

git allows independent configurations between line endings used when a developer checkouts out code locally, and when code is committed to a repository.
IDEs are often better suited for different line endings, with LF line endings becoming more and more standardised.
However, NVDA is Windows based development, where CRLF line ending usage is commonly default or compulsory.
Developers can checkout code locally, using whatever line endings they prefer or their IDE requires, using git config.core autocrlf.

We can configure what line endings we commit to the repository with .gitattributes.
This is set on a repository level.
As what line endings we commit with is agnostic to what is checked out, the decision is based on what works best for the repository.
Unix style line endings usage and support continues to increase on Windows and LF files.
LF tends to be better supported by Unix style developer tools such as git.
LF uses less data than CRLF.

Additional information: https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings

Description of how this pull request fixes the issue:

Fixing this is a 3 step process, this PR emulates the 3 steps, but only one commit should be merged at a time.

  1. The first commit updates .gitattributes so that any new files committed will use LF style endings where appropriate.
  2. Then, no PRs should be merged until a commit which normalizes the line endings is committed to master.
    Otherwise, new PRs will have large diffs due to line ending changes.
    git pull origin/master # to get .gitattributes
    git add --renormalize .
    git commit -m "fix line endings"
  3. Then, the third commit can be committed. This updates the .git-blame-ignore-revs file so that the normalization commit can be ignored by developers using git blame.

Specifically this PR:

  • Updates .gitattributes file, with eol=lf and normalization patterns for each file type NVDA uses:
    • -text or binary for no normalization
    • text to change line endings to lf when committing code.
  • Ensures git add --renormalize . sets the correct line endings for the entire project.
  • Updates documentation

Testing strategy:

Setup normalize-line-endings as follows to perform this testing.

git checkout normalize-line-endings
git reset origin/normalize-line-endings --hard
git add --renormalize .
git commit -m "fix line endings"

Update .git-blame-ignore-revs with the latest commit.

Compare the diff while ignoring eol line changes

git diff --ignore-cr-at-eol origin/master normalize-line-endings

Check the blame of a file from this branch and confirm the line ending changes show up in the blame.

Then check the blame of a file:

git blame source/core.py normalize-line-endings

Test that .git-blame-ignore-revs works and ignores the line ending changes.

git blame --ignore-revs-file .git-blame-ignore-revssource/core.py normalize-line-endings

Test that setting .git-blame-ignore-revs works.

This should be recommended to devs.

git config --local blame.ignoreRevsFile .git-blame-ignore-revs
# should ignore revs
git blame source/core.py normalize-line-endings
# should not ignore revs
git blame --ignore-revs-file "" source/core.py normalize-line-endings

Test that committing a text file, not used by the translation system, with CRLF line endings is not possible.

This can be done by adding or changing a file.

Test that the resulting translation .pot files generated do not change after nomalization.

git checkout master
scons pot
git checkout normalize-line-endings
scons pot

Note: Not git diff, as these files are ignored. Update the commits within the file names if required.

diff output/nvda_snapshot_source-master-6b4cf45.pot output/nvda_snapshot_source-normalize-line-endings-31dee0a.pot

cat -A displays CR as ^M. This will ensure the diff tool picks up on any CRLF changes.

cat -A output/nvda_snapshot_source-master-6b4cf45.pot > master-with-explicit-line-endings.pot
cat -A output/nvda_snapshot_source-normalize-line-endings-31dee0a.pot > normalized-with-explicit-line-endings.pot
diff master-with-explicit-line-endings.pot normalized-with-explicit-line-endings.pot

Known issues with pull request:

None

Change log entries:

N/A

Code Review Checklist:

  • Pull Request description is up to date.
  • Unit tests.
  • System (end to end) tests.
  • Manual testing.
  • User Documentation.
  • Change log entry.
  • Context sensitive help for GUI changes.
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers

Summary by CodeRabbit

  • Documentation

    • Added detailed usage guidelines for the git-blame-ignore-revs feature.
    • Updated codingStandards.md with new file encoding and line ending practices, naming conventions, and docstring format specifications.
  • Chores

    • Modified .gitattributes to set eol=lf, define text attributes for various file types, and specify files exempt from renormalization.

@seanbudd seanbudd requested a review from a team as a code owner September 8, 2021 08:23
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit cbce460fa3

@seanbudd seanbudd mentioned this pull request Sep 8, 2021
5 tasks
@LeonarddeR

Copy link
Copy Markdown
Collaborator

Git for Windows has a feature to fiddle with line endings at checkout and commit time I believe. Won't this conflict with that?

seanbudd added a commit that referenced this pull request Sep 8, 2021
seanbudd added a commit that referenced this pull request Sep 8, 2021
@seanbudd seanbudd force-pushed the normalize-line-endings branch from f6da480 to 123c3ae Compare September 8, 2021 08:58
@seanbudd

seanbudd commented Sep 8, 2021

Copy link
Copy Markdown
Member Author

Git for Windows has a feature to fiddle with line endings at checkout and commit time I believe. Won't this conflict with that?

This should remain compatible with core.autocrlf. See the documentation changes to devDocs/codingStandards.md.

See also: https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings

@CyrilleB79

Copy link
Copy Markdown
Contributor

Did you update the readme.md with the recommanded configuration? Sorry, I've not been able to find it in a such big number of modified files because I do not know how to view a specific modified file in GitHub.

@seanbudd

seanbudd commented Sep 9, 2021

Copy link
Copy Markdown
Member Author

Did you update the readme.md with the recommanded configuration?

There's no real recommended configuration for config.core autocrlf - the decision on what value to use is up to the developer, a preference which might be influenced by IDE support. The .gitattrbutes file should ensure line endings are committed properly, and config.core autocrlf configures how code is checked out locally. More information.
I've updated devDocs/codingStandards.md to reflect this, it's not mentioned in the readme currently.

I plan on adding git config --local blame.ignoreRevsFile .gitignorerevs to the Contributing wiki page, as that's where the git setup is mentioned. I'm not aware of any other good places to mention it.

Sorry, I've not been able to find it in a such big number of modified files because I do not know how to view a specific modified file in GitHub.

Yes this PR is a proof of concept on the strategy. I think each commit/step should be merged as separate PRs. The last and the first commits of this PR are the ones worth manually checking. Hiding whitespace changes via the diff settings makes browsing this whole diff a bit more manageable.

@feerrenrut

This comment has been minimized.

@lukaszgo1

This comment has been minimized.

@feerrenrut

This comment has been minimized.

@seanbudd

This comment has been minimized.

seanbudd added a commit that referenced this pull request Sep 20, 2021
@seanbudd seanbudd force-pushed the normalize-line-endings branch from 123c3ae to cb81c5d Compare September 20, 2021 05:21
seanbudd added a commit that referenced this pull request Sep 20, 2021
@seanbudd seanbudd force-pushed the normalize-line-endings branch from cb81c5d to a5810c7 Compare September 20, 2021 05:41
@seanbudd seanbudd changed the title Normalize line endings Update git attributes to prefer LF Sep 20, 2021
@seanbudd

Copy link
Copy Markdown
Member Author

@feerrenrut @michaelDCurran Any ideas on how we can confirm that normalizing the line endings won't affect translators?

Otherwise, this PR is ready to be reviewed then merged with the strategy outlined in the PR description.

@michaelDCurran

This comment has been minimized.

@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit eb6e6aa1aa

@seanbudd

Copy link
Copy Markdown
Member Author

Thanks @michaelDCurran - I've updated the testing steps and confirmed that the pot file is unchanged (other than the metadata eg commit info and generation time).

@AppVeyorBot

This comment has been minimized.

@feerrenrut

Copy link
Copy Markdown
Contributor

@seanbudd When you do that diff of the pot files, ensure that the line endings were considered and not ignored by the diff tool. It may be more certain to inspect an entry in the file manually with tail file.pot | cat -A.

@feerrenrut feerrenrut left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope you don't mind I've updated the description to add a numbered lists for the stages of the change. It currently talks about 3 commits in this PR which is a little confusing.

How will the PR's be updated to resolve the diff? Each PR will have to have normalize run on it and a new commit pushed, right?

@seanbudd

Copy link
Copy Markdown
Member Author

@feerrenrut

When you do that diff of the pot files, ensure that the line endings were considered and not ignored by the diff tool. It may be more certain to inspect an entry in the file manually with tail file.pot | cat -A.

I've confirmed this and updated the testing steps.

I hope you don't mind I've updated the description to add a numbered lists for the stages of the change. It currently talks about 3 commits in this PR which is a little confusing.

Thanks, this is much easier to read.

How will the PR's be updated to resolve the diff? Each PR will have to have normalize run on it and a new commit pushed, right?

I've created #12864 to investigate and discuss this. In short - yes.

@seanbudd seanbudd marked this pull request as draft September 21, 2021 02:27
@seanbudd seanbudd changed the title Update git attributes to prefer LF Update .gitattributes to prefer LF Sep 23, 2021
@seanbudd seanbudd added this to the 2024.3 milestone Jun 7, 2024
@seanbudd seanbudd added the merge-early Merge Early in a developer cycle label Jun 17, 2024
@seanbudd seanbudd force-pushed the normalize-line-endings branch from 62b2420 to 46c0bee Compare June 27, 2024 02:18
@seanbudd seanbudd marked this pull request as ready for review June 27, 2024 02:19
@coderabbitai

coderabbitai Bot commented Jun 27, 2024

Copy link
Copy Markdown
Contributor

Walkthrough

The update introduces enhanced documentation and guidelines for using the git-blame-ignore-revs feature, standardizes file attributes and encoding standards in .gitattributes, and refines coding standards in codingStandards.md with a focus on consistency in file encoding, indentation, naming conventions, and documentation formats.

Changes

Files/Groups Summary
.git-blame-ignore-revs Added documentation and usage guidelines for the git-blame-ignore-revs feature.
.gitattributes Defined eol=lf, text attributes for multiple file types, specified files exempt from renormalization, etc.
projectDocs/dev/codingStandards.md Updated guidelines for UTF-8 encoding, LF line endings, naming conventions, tabs for indentation, and Sphinx docstring format.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@seanbudd seanbudd added the release/blocking this issue blocks the milestone release label Jun 27, 2024

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
Comment thread projectDocs/dev/codingStandards.md Outdated
@seanbudd seanbudd requested a review from a team as a code owner June 27, 2024 02:28
@seanbudd seanbudd requested a review from Qchristensen June 27, 2024 02:28
@seanbudd seanbudd force-pushed the normalize-line-endings branch from 6167ba7 to 3fb0c8c Compare June 27, 2024 02:32
@seanbudd seanbudd force-pushed the normalize-line-endings branch from 3fb0c8c to ff61724 Compare June 27, 2024 02:40
@seanbudd

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Outside diff range and nitpick comments (1)
projectDocs/dev/codingStandards.md (1)

Line range hint 21-86: Refine and approve the guidelines on indentation and identifier names.

The guidelines on indentation and identifier names are well-defined and promote readability and maintainability. However, the repetition of the word 'should' in the section on Enums could be addressed to enhance readability.

- Enums should be formatted using the expected mix of above eg:
+ Enums should be formatted using the expected mix of the above guidelines, for example:
Tools
LanguageTool

[duplication] ~21-~21: Possible typo: you repeated a word
Context: ...t-Configuration#_core_autocrlf).

Indentation

  • Indentation must be done with tabs (one per level),...

(ENGLISH_WORD_REPEAT_RULE)

Comment thread projectDocs/dev/codingStandards.md
Comment thread projectDocs/dev/codingStandards.md Outdated
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit be214c00b5

@seanbudd seanbudd merged commit 9ea2566 into master Jun 27, 2024
@seanbudd seanbudd deleted the normalize-line-endings branch June 27, 2024 06:07
seanbudd added a commit that referenced this pull request Jun 27, 2024
Follow up to #12816
Closes #12387

Must be merge commit

Performed the following:

git add --renormalize .
git commit -m "normalize line endings"
Added the commit to .git-blame-ignore-revs

Testing

Confirmed testing by ensuring the diff is empty

git diff --ignore-cr-at-eol origin/master normalize-all-line-endings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-early Merge Early in a developer cycle release/blocking this issue blocks the milestone release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants