Note: it would be nice to also ensure that we don't print nonsensical code for larger source files -- we could either disable the fallback of re-reading the source completely, or try to disable it conditionally when lexer directives are used. This is sensibly more work than the current workaround, so I think a separate PR would be better.

You know you're getting desperate when you increase the size of buffers to make a bug go away :-)

If you really want to go this way, I'd suggest reading the source code into a string (In_channel.input_all) then run the lexer over this string (Lexing.from_string). At least, the memory usage will be proportional to the size of the source code, instead of being absurdly high for tiny source files.

(For what it's worth: after much back and forth, the CompCert compiler also settled on reading the preprocessed C source code into a string before lexing and parsing, although I can't remember all the reasons why; but there were several.)

I also considered this approach, but I was somewhat worried about the idea of unbounded memory usage for files that are larger than 256Kib. In the compiler we have a menhir-generated parser.ml file clocking at 2.4MiB. Do we want a cutoff based on the file size, or are we happy with using a lot of memory on very large files? (As discussed in #12238 (comment), very large files also produce large build artifacts, so I guess their live memory usage is high anyway, but the build artifacts seem to remain smaller than the source code in practice.)

Do we want a cutoff based on the file size, or are we happy with using a lot of memory on very large files?

My intuition is that the compiler will need much more than X of RAM to process a file of size X (and esp. so on 64-bit platforms); the extra cost of keeping the full buffer in RAM seems definitely ok to me.

My intuition is that the compiler will need much more than X of RAM to process a file of size X

Yes, this is certainly true in general. Out of curiosity, I added code to the compiler to print out the file size and typedtree size (based on Obj.reachable_words) during compilation. A few examples, picked at random:

option.ml: file is 2104 bytes; typedtree is 530968 bytes (252x larger)
lambda/translcore.ml: file is 47477 bytes; typedtree is 4906824 bytes (103x larger)
typing/includemod.ml: file is 46043 bytes; typedtree is 5687624 bytes (124x larger)

It's pretty clear that reading the whole file into memory for lexing is unlikely to cause memory usage problems (except on improbably large files consisting mostly of whitespace and comments, I suppose).

Thanks! I will try to write a follow-up PR that does this.

In #12403 I propose to read the whole source file in one go before lexing.