Skip to content

Add basic support for localization#4729

Merged
bk2204 merged 9 commits intogit-lfs:mainfrom
bk2204:locale
Nov 15, 2021
Merged

Add basic support for localization#4729
bk2204 merged 9 commits intogit-lfs:mainfrom
bk2204:locale

Conversation

@bk2204
Copy link
Member

@bk2204 bk2204 commented Nov 9, 2021

Most people on the planet do not speak English. We'd like Git LFS to be accessible to them as well, so add basic support for localization. Each commit explains its rationale, but this introduces a localization framework and but a single localized string. However, this allows us to continue localizing more and more of Git LFS and permits translators to start when they're comfortable.

@bk2204 bk2204 force-pushed the locale branch 2 times, most recently from 6a7e486 to 7b4a06d Compare November 9, 2021 21:47
We'd like to support localization in Git LFS.  To do so, add the gotext
package, which implement gettext-compatible localization.
We'd like to support localizing Git LFS.  We have the gotext package,
which supports several different ways of getting GNU gettext-compatible
locale strings.  However, we need some central message catalog we can
use.

Let's create a singleton by the name of Tr.  That means that we can
translate a string by writing something like this:

  fmt.Println(tr.Tr.Get("This is a message about %s.", subject))

These strings can be automatically extracted by the xgotext tool
available from the package
github.com/leonelquinteros/gotext/cli/xgotext.  While currently quite
slow, it does understand Go syntax, unlike the standard GNU gettext
suite.

We initialize it based on the locale.  Since Go does not use the
standard C library, we have to emulate its behavior, as outlined in
locale(7).  If our locale is "en_DK.UTF-8", we first look for "en_DK"
and then "en", in that order.  This preserves a fallback for languages
which are similar across most speakers, like English and Spanish, and
can share a translation, but for locales which cannot, such pt_PT and
pt_BR, or zh_TW and zh_CN, this ensures we prefer a more appropriate
locale.

Introduce a variable called "locales" which will be used to store
Base64-encoded MO files (compiled Gettext files) for each locale.  We
can still load external locales from the standard directory
(/usr/share/locale), but this preserves the behavior of providing a
single, relocatable binary, which people like.  If this becomes a
problem in the future, we can revisit it.  Population of this variable
will come in a future commit.
Currently, Git LFS has the benefit of being a single, relocatable
binary.  While it is customary to store locales in /usr/share/locales on
Unix systems, this isn't necessarily the case on Windows, and we'd like
to preserve the ability of users to move the binary where they like.

As a result, let's add a program which can be run by go generate that
embeds these translations into the binary.  They are Base64-encoded to
make handling them a little easier and avoiding need to write giant
strings of backslashed escape sequences.  The reader will note that this
is based off the related command to embed the manual pages.
As the first thing we do when we run a command, let's initialize the
locale.  This will let us print any sort of error or warning messages in
the user's preferred language if we have any problems.
@bk2204 bk2204 marked this pull request as ready for review November 10, 2021 20:24
@bk2204 bk2204 requested a review from a team November 10, 2021 20:24
Copy link
Member

@chrisd8088 chrisd8088 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great stuff, thank you!


//go:generate go run ../tr/trgen/trgen.go

var Tr = gotext.NewLocale("/usr/share/locale", "en")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of interest, how will this operate on Windows?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will not, but since we don't expect external locale files at all for Git LFS, it shouldn't be a problem. It may be that a Linux distro chooses to ship these externally, but that wouldn't work on Windows anyway since Windows doesn't have a canonical place for them, so again, the fact that this does nothing shouldn't be a problem.

Let's make sure we build the translations into the binary when possible.
Specifically, if the msgfmt command exists, let's compile the .po files
into .mo files and then embed their contents into the binary using the
generator command to do that.

Let's also ignore the built .mo files and the generated .pot template
files so we don't check them in.
It will take some time for us to get the codebase translated.  However,
as a test of our codepaths and as a demonstration of what to do, let's
introduce a single translation, plus a simple Spanish translation that
translates only that string.  This allows an invocation like the
following to print a simple translated message:

  LC_ALL=es_ES.UTF-8 git lfs filter-process
When building the default image in CI, or a release build, ensure that
the gettext package is installed so we can test locales.  We don't
include it in all CI runs to ensure that we also test building without
localization.

Note that on Windows, these tools should always be installed as part of
the Git for Windows environment, since they are needed for Git, and we
would have no way to install them anyway.
@bk2204 bk2204 merged commit 47efc94 into git-lfs:main Nov 15, 2021
@bk2204 bk2204 deleted the locale branch November 15, 2021 17:25
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Dec 10, 2024
Since commit 6285716 in PR git-lfs#4729 we
have installed the gettext library and utilities in our CI and release
GitHub Actions jobs, on both macOS and Ubuntu Linux platforms, while
on Windows we rely on the library and utilities being installed in
advance as part of the Git for Windows environment.

At present, though, the macOS runners provided by GitHub Actions are
always provisioned with a pre-installed version of a recent release of
the gettext library, as packaged by Homebrew, which includes the
msgfmt(1) program used by our Makefile, so we no longer need to install
the "gettext" Homebrew formula ourselves.
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Dec 15, 2024
Since commit 6285716 in PR git-lfs#4729 we
have installed the gettext library and utilities in our CI and release
GitHub Actions jobs, on both macOS and Ubuntu Linux platforms, while
on Windows we rely on the library and utilities being installed in
advance as part of the Git for Windows environment.

At present, though, the macOS runners provided by GitHub Actions are
always provisioned with a pre-installed version of a recent release of
the gettext library, as packaged by Homebrew, which includes the
msgfmt(1) program used by our Makefile, so we no longer need to install
the "gettext" Homebrew formula ourselves.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants