This is a modest change, it makes mandating UTF8 less-compatibility-breaking, and it solves a practical problem with the maintenance of existing non-English textbooks that relied on latin1 encoding. I must say that I like the idea of starting with (something like) this as a baby step towards proper Unicode identifiers.

Reading the code again, I think it can be made a bit nicer (and avoid the big regexps) by using some UTF8 functions that were introduced in OCaml 4.14.

In my mind, it's not clear what we should shoot for: a quick hack like this or a full implementation of Unicode identifiers as in UAX 31 and @whitequark 's old prototype . Opinions welcome.

I think solving this would be a first good step to normalise the situation. I suspect most people like me who write end-user apps in OCaml for non-english speaking users or long form documentation in .mli and .mld files moved to UTF-8 encoded files a long time ago.

I note however that this does not check that files are UTF-8 encoded which would be good to have.

A few comments.

I felt obliged to normalize the UTF-8 encoding of accented letters (I chose NFD)

If the compiler does use String.capitalize_ascii or Char.uppercase in a meaningful way NFD may actually be a requirement rather than a choice unless a Unicode aware case mapping is added.

Which brings the question… what happens with the file éléphant.ml in the room ? @whitequark has a section about file systems. You likely need to normalise the filenames of compilation units. Only Apple's file systems seem to guarantee you NFD. Other systems do nothing about it and likely give you NFC (the output of input methods).

Other than that, except for size, and possible expectations in other tooling downstream (debuggers, profilers) I think the choice between NFD or NFC is mostly irrelevant.

because different text editors save Unicode text in different forms: e.g. NFC for Emacs and most Linux utilities, and NFD for xcode, textedit, and everything made by Apple.

I'm surprised by this. I tried here and both TextEdit and Xcode give me NFC with a CH-FR input method.

In my mind, it's not clear what we should shoot for: a quick hack like this or a full implementation of Unicode identifiers as in UAX 31 and @whitequark 's old prototype . Opinions welcome.

Moving to checked UTF-8 encoded files with a compatibility story for latin1 users looks like a low hanging fruit that needs little Unicode machinery. I would rather aim for that first.

Thanks for the comments @dbuenzli, your points are well taken. I remembered wrongly about macOS text editors favoring NFD over NFC. Concerning interactions with files (the éléphant in the pièce), @whitequark 's analysis is interesting, but I need to run more experiments too.

I strongly recommend NFC. IIUC NFD is widely considered to be a mistake.

I strongly recommend NFC. IIUC NFD is widely considered to be a mistake.

Would you care to elaborate ?

Without proper out-of-band agreements, you have to be prepared for any, even mixed, forms on foreign input.

After that it's up to you to convert and use the form that is the easiest to work with for the task at hand internally.

There's no such thing as a "mistaken" normal form. However there may be mistaken programs making assumptions they shouldn't do on foreign inputs :-)

I strongly recommend NFC. IIUC NFD is widely considered to be a mistake.

Would you care to elaborate ?

Without proper out-of-band agreements, you have to be prepared for any, even mixed, forms on foreign input.

After that it's up to you to convert and use the form that is the easiest to work with for the task at hand internally.

There's no such thing as a "mistaken" normal form. However there may be mistaken programs making assumptions they shouldn't do on foreign inputs :-)

Sorry, I meant that Apple’s use of NFD is considered to be a mistake, IIUC

Thanks for the feedback. The next iteration is here: #11736 . It does use NFC instead of NFD, and tries to address the éléphant.ml problem.