I guess @damiendoligez summons me so that I can express my usual concern about this topic.

Comments are intended first and foremost for writing prose in natural language. Commenting code out comes second. "Lexing" the contents of comments using the same lexical rules as for OCaml code might improve the "commenting out" behavior, but should not complicate the writing of natural language.

Here, I think @damiendoligez has counterexamples showing that lexing identifiers is not enough, and I would like to be reassured that normal uses of single quotes in text is not impacted.

I agree that commenting out code is not what comments are intended for, but comments are used for documentation too, and thus can contain code. In any case, the current situation is that the content of comments is checked for strings and character literals, which maybe already complicates things more than it should, especially if it doesn't even work. I think this simple fix makes it work in all cases without complicating things further, but maybe I'm missing something.

The only use of single quotes in natural language that is affected, is when the sequence '"' (single-double-single) occurs, and even then it's only a problem in certain combinations like a'"' or a' '"'.

I stumbled upon it while trying to let Atom highlight comments correctly, which turns out to be rather difficult, and I don't think any editor does it right (but I'm getting there).

I don't think any editor does it right

Emacs (caml-font.el) did, at some point before the lexing of number literals was simplified. It should be easier now.

Give me a few days to think about completeness and check this against OPAM packages.

Side note: I almost wish that there was a distinct syntax for commenting out blocks of code, because I sometimes get myself in trouble when I include double quotes and other such things in comments. The fact that comments aren't pure uninterpreted blocks of text is a double edged sword. It makes it much easier to comment out code, but it makes the behavior of comments as documentation a bit more unexpected.

I almost wish that there was a distinct syntax

Note especially lightweight, but you can use quoted strings:

(*{foo|
      blabl"abla
   |foo}*)

Note especially lightweight, but you can use quoted strings:

I suppose you can, though you need caution if commenting out the last part of a function. More to the point, though, this doesn't fix the fact that a comment like:

(* recognizes: ' " @ + *)

is unexpectedly hard to write.

I think Alain's point is that you just write:

(*{| recognizes: ' " @ + |}*)

I.e. this is a "special syntax" for free-form comments, not for commented code.

I suppose you can, though you need caution if commenting out the last part of a function.

What do you mean?

More to the point, though, this doesn't fix the fact that a comment like:

(*{|
recognizes: ' " @ +
|}*)

Not especially lightweight, but you can use quoted strings

This doesn't work for docstrings. Something like "quoted comments" (e.g. (*foo| recognizes: ' " @ + |foo*)) would work, but it's too late for that.

What do you mean?

I mean that if one is naively writing comments and is used to other languages, one believes that the contents of a comment are free text, but they aren't. You can't just write whatever you feel like inside them. If there was a distinct way to comment out code blocks, this would not have to be the case. But that's not going to happen so it's just me musing.

Emacs (caml-font.el) did, at some point before the lexing of number literals was simplified.

I only tried tuareg mode, which does it wrong. caml-font.el handles comments correctly, but is worse than tuareg for other things.

@314eter Possibly a good reason to file a report with the Tuareg team, pointing them at the code that works correctly.

The same problem occurs in ocamllex, where something like

rule token = parse _ { f' '"' }

gives an error, and it can be solved the same way in lex/lexer.mll.

Can I include that in this PR, or is it better to open a new one?

I think it's better to open a new PR for ocamllex. This is a language change, while for ocamllex it's a bug fix.

I have no experience with emacs and/or lisp, so I'll probably mess something up if I try to update caml-font.el.

Fixing caml-font.el was not as difficult as I thought.

Fixing caml-font.el was not as difficult as I thought.

Would similar fixes to the tuareg code be straightforward?

@damiendoligez Do you have time to take a look at this (and maybe #1932) again?

I removed caml-font.el and opened ocaml/caml-mode#4 instead.

I rebased.

Merged, thanks!

In ocaml/opam-repository#16114 (comment) the current maintainer of pfff (cc @aryx) did hit an issue that seems to have been carried by this change in the lexer.

For example, comparing the same code with OCaml 4.09.0 and 4.10.0, we get a different result:

        OCaml version 4.09.0

# type t = Test of unit (* '"' or b'"' *) * unit list * unit (* '"' *);;
type t = Test of unit * unit list * unit

vs.

        OCaml version 4.10.0

# type t = Test of unit (* '"' or b'"' *) * unit list * unit (* '"' *);;
type t = Test of unit

Is this change intended?

That change is intended, since comments are used to comment out OCaml code or to write OCaml code in documentation. In OCaml, b' would be an identifier, followed by the string "' *)...(* '".

Unfortunately, this breaks in other languages like PHP. I don't know why it didn't fail to build when @damiendoligez tested this change on opam (#1901 (review)).

I don't know why it didn't fail to build when @damiendoligez tested this change on opam (#1901 (review)).

The latest version of pfff that was available on opam did not support OCaml >= 4.09, so this particular one wasn't tested.