Hm, I thought the test failures were unrelated when I ran them locally (other than the one I fixed in the PR), but several of them do look to be related to the # printing change. I'll take a look.

(The PR changes whether # is printed when stdin isn't a tty, since libedit doesn't print it, so I changed it even when libedit is disabled for consistency, and since it seems superfluous anyways.)

OK, issues worked out, everything passing on Travis now :)

Ping! What needs to happen to get this merged?

I'm personally reluctant to invest time on change improving the ocaml toplevel, given that utop is better for end-user in all possible ways, and I think we should rather encourage people to use it than to bear with the default one longer. (ocaml is useful for compatibility, during compiler hacking for quick tests, and serves as a good way to test for dynamic stuff.)

That said, this change seems very small (except the humongous testsuite change, which I'm not sure is really a good idea; couldn't we just always print the # when TERM=dumb is set?).

I'm personally reluctant to invest time on change improving the ocaml toplevel, given that utop is better for end-user in all possible ways, and I think we should rather encourage people to use it than to bear with the default one longer.

Sure, but until/unless utop or similar becomes the default actually shipped with the main distribution, the current toplevel is still the default, always-available option, and seems worth making usable if it isn't going to be immediately replaced. Defaults matter, and ocaml, not utop, is the default.

except the humongous testsuite change, which I'm not sure is really a good idea; couldn't we just always print the # when TERM=dumb is set?

If you feel strongly, I can reverse it, but the behavior in this PR seems like the right thing to do. If you look at some of the changed output/expect files, the spurious prompts get in the way of reading them a lot. The prompt is not always just the # -- there's tons of whitespace and other characters (mostly *) too. That stuff only makes sense and is usable if you're an actual user sitting at a terminal.

On 25 Sep 2015, at 23:33, Josh Watzman notifications@github.com wrote:

I'm personally reluctant to invest time on change improving the ocaml toplevel, given that utop is better for end-user in all possible ways, and I think we should rather encourage people to use it than to bear with the default one longer.

Sure, but until/unless utop or similar becomes the default actually shipped with the main distribution, the current toplevel is still the default, always-available option, and seems worth making usable if it isn't going to be immediately replaced. Defaults matter, and ocaml, not utop, is the default.

Very true... there are very few pointers to beginners to find utop at the moment, and a common reported problem from Real World OCaml readers is that they are using the normal ocaml toplevel (despite clear installation instructions, which few people fully read of course).

-anil

I would personally be more inclined to invest work toward "making utop the default". I think this could and should happen outside what is called the "compiler distribution", which is an entity defined by maintenance aspects rather than "what we advise beginners to use by default" (Menhir is a much better default than OCamlyacc for example, and I say it everytime I have the occasion to do so).

I don't want to keep @jwatzman 's patch (which I suspect is fine) hostage of this discussion, so I will try to find another channel for it. If anyone has time to review and maybe merge the proposed change, please do!

(No, no strong feelings about #, I haven't had time to think about it so I would rather trust your judgment.)

More comments are welcome on the "making utop the default instead" issue.

A technical comment on this PR: in the fallback case, the mixture of stdio.h functions and Unix-level functions (e.g. read()) is suspect. In particular, I'm pretty sure that feof() returns the correct result only if the last read from stdin was done using stdio.h functions, which is not the case here.

A question: is libedit working and available under Windows?

(Edit: mark-up was garbled; fixed it.)

I was about to comment on Windows. As far as I know, in my current
setup, opam install utop doesn't work (build failures in lwt). utop has
non-trivial dependencies; adding editline support to ocaml (and making
sure it works on windows) would be a good interim solution until
everyone fixes their build system and "opam install utop" works out of
the box on windows.

There seems to be a windows port of editline
http://thrysoee.dk/editline/. I could try to build your patch with
that library on windows and see if it works.

~ jonathan

On 09/27/2015 09:06 AM, Xavier Leroy wrote:

More comments are welcome on the "making utop the default instead" issue.

A technical comment on this PR: in the fallback case, the mixture of
functions and Unix-level functions (e.g. read()) is suspect. In
particular, I'm pretty sure that feof() returns the correct result
only if the last read from stdin was done using functions, which is
not the case here.

A question: is libedit working and available under Windows?

—
Reply to this email directly or view it on GitHub
#241 (comment).

One possibility is to make the opam switch installation install utop by
default. I'm always confused when I install a new compiler version and
don't have utop or Merlin.

On Sep 27, 2015, at 12:39 PM, Jonathan Protzenko notifications@github.com
wrote:

I was about to comment on Windows. As far as I know, in my current
setup, opam install utop doesn't work (build failures in lwt). utop has
non-trivial dependencies; adding editline support to ocaml (and making
sure it works on windows) would be a good interim solution until
everyone fixes their build system and "opam install utop" works out of
the box on windows.

There seems to be a windows port of editline
http://thrysoee.dk/editline/. I could try to build your patch with
that library on windows and see if it works.

~ jonathan

On 09/27/2015 09:06 AM, Xavier Leroy wrote:

More comments are welcome on the "making utop the default instead" issue.

A technical comment on this PR: in the fallback case, the mixture of
functions and Unix-level functions (e.g. read()) is suspect. In
particular, I'm pretty sure that feof() returns the correct result
only if the last read from stdin was done using functions, which is
not the case here.

A question: is libedit working and available under Windows?

—
Reply to this email directly or view it on GitHub
#241 (comment).

—
Reply to this email directly or view it on GitHub
#241 (comment).

One possibility is to make the opam switch installation install utop by default.

Some people use opam for deployment, that wouldn't make sense at all.

On Sun, Sep 27, 2015 at 12:53 PM, Gabriel Radanne notifications@github.com
wrote:

One possibility is to make the opam switch installation install utop by
default.

Some people use opam for deployment, that wouldn't make sense at all.

This sounds related to the idea of a "platform", which can be thought of as
just a collection of tools and libraries that is officially recommended.

the mixture of stdio.h functions and Unix-level functions (e.g. read()) is suspect

Oops, good catch, will update PR later to fix. (As well as rebasing it.)

A question: is libedit working and available under Windows?

I have no idea. My expectation was that either it would be available and picked up, or not and the status quo would be maintained.

There seems to be a windows port of editline http://thrysoee.dk/editline/. I could try to build your patch with that library on windows and see if it works.

@protz yes, I'd appreciate that, I don't have a Windows build environment to test with.

More comments are welcome on the "making utop the default instead" issue.

It has a few dependencies,

> opam list --required-by utop --rec 
# Available packages recursively required by utop.1.18 for 4.02.3:
base-bytes      base  Bytes library distributed with the OCaml compiler
base-threads    base  Threads library distributed with the OCaml compiler
base-unix       base  Unix library distributed with the OCaml compiler
camlp4        4.02+6  Camlp4 is a system for writing extensible parsers for prog
camomile       0.8.5  A comprehensive Unicode library
cppo           1.3.0  Equivalent of the C preprocessor for OCaml programs
lambda-term      1.9  Terminal manipulation library for OCaml
lwt            2.5.0  A cooperative threads library for OCaml
ocamlfind      1.5.5  A library manager for OCaml
ppx_tools     0.99.2  Tools for authors of ppx rewriters and other syntactic too
react          1.2.0  Declarative events and signals for OCaml
zed              1.4  Abstract engine for text edition in OCaml

While I would welcome completion and line edit in the distributed ocaml toplevel, I can't cope with all the visual noise of utop --- I personally don't need to have a completion line while I'm not trying to complete anything, and I don't need to know the time and number of evaluations done so far after each phrase inserted. So at least if utop is made the default please make it reasonable from a UI point of view.

If you feel strongly, I can reverse it, but the behavior in this PR seems like the right thing to do. If you look at some of the changed output/expect files, the spurious prompts get in the way of reading them a lot. The prompt is not always just the # -- there's tons of whitespace and other characters (mostly *) too. That stuff only makes sense and is usable if you're an actual user sitting at a terminal.

Note that when you run ocaml in emacs with the tuareg mode, TERM is set to dumb. So if you don't display # then users won't get a prompt. Plus ocaml already has a -noprompt option, so we can use this instead for tests.

2015-09-27 19:32 GMT-04:00 Daniel Bünzli notifications@github.com:

While I would welcome completion and line edit in the distributed ocaml toplevel, I can't cope with all the visual noise of utop --- I personally don't need to have a completion line while I'm not trying to complete anything, and I don't need to know the time and number of evaluations done so far after each phrase inserted. So at least if utop is made the default please make it reasonable from a UI point of view.

I don't know how to get rid of the completion bar but you can get a
"decent" prompt by adding the following to your .ocamlinit:

#require "lambda-term";;
#require "react";;
UTop.prompt := React.S.const (LTerm_text.of_string "# ");;

@trefis, you can also use #utop_prompt_dummy;; for the same effect. For the completion bar you can use UTop.set_show_box false, but then you loose completion entirely... I need to add an option for classic-style completion.

@jwatzman

Sure, but until/unless utop or similar becomes the default actually shipped with the main distribution, [...]

When you say "the main distribution", you should be thinking of OPAM rather than the source tar file published by Inria, which is on a trend of containing less and less stuff beyond the compiler.

@agarwal

This sounds related to the idea of a "platform", which can be thought of as
just a collection of tools and libraries that is officially recommended.

Not just officially recommended, but also with a nice easy-to-use installer. How do we make it happen?

[sorry for the close, I pressed the wrong button]

Not just officially recommended, but also with a nice easy-to-use installer. How do we make it happen?

The Platform is an OCaml Labs project. I don't know more.

I just pushed a new update which

• rebases on current trunk
• fixes the usage of feof
• fixes some inconsistencies in when the prompt is printed and related behavior -- basically, it now falls back on the old behavior if it isn't sure you're using an interactive tty
• splits out the test suite change

Windows testing of this would be useful -- I don't know if it even compiles there, I don't have a Windows build environment to test. The only thing that might not work there is the call to isatty, internet searches were unclear if it needs to be spelled _isatty there, but it does seem to exist?

When you say "the main distribution", you should be thinking of OPAM rather than the source tar file published by Inria, which is on a trend of containing less and less stuff beyond the compiler.

Then Inria / the ocaml core devs needs to take actual steps to make sure that people think of it that way!

Right now, here's what new users see. They hear there's this cool language called "ocaml", and oh nice, there's a package already for their OS for it, which they install with apt-get install ocaml or brew install ocaml or whatever. And cool, there is a binary called ocaml in there, which is probably a nice REPL like Python or Haskell provide, and yeah, if you run it, that's what it is, cool! But oops, it sucks, basic functionality like the up arrow doesn't even work, and now they have a bad first impression.

It's not until they get to the point of extensive googling about ocaml and tutorials that they see what they really should have done was download a separate package manager (opam) and installed the compiler through that, and then used it to install some other weird thing called "utop" and used that instead of the deprecated "ocaml" binary which is deprecated despite it being named after the language!

And googling for a tutorial might not even help depending on where you land; the this is the top google hit for "ocaml tutorial", of which the "basics" link goes to a page which recommends the ocaml toplevel. And even if you could fix the top N tutorials and books to recommend utop, its fairly complicated install process (i.e., opam) is going to turn people off unless you can package it as something provided by apt-get install ocaml. (Edit to clarify: no opam is not particularly hard to install, but it's harder than the familiar apt-get install ocaml, considerably so when you account for needing the knowledge that you even need something called "opam" in the first place!)

Bootstrapping this knowledge is hard, even for someone more experienced in the language, and the current status quo does not push people into the right direction or towards building the right mental model of the way the ecosystem works under the opam ideal.

Sorry to rant a bit -- I just wanted to elaborate on why "well just use utop" is not really a good solution to basic usability problems with an important, front-and-center piece of the ocaml suite -- and why "Inria's tarball is not canonical" is totally the wrong way to look at this from an end-user perspective.

I more than agree with your point above. We should not say "just use utop", but also make sure there is enough documentation for end-users to find on their natural path to OCaml programming. The remarks you make on the current state of the information ecosystem are very useful, because they point precisely at what can and should be fixed.

We only get to be a beginner once, but I've had on several occasions the opportunity of watching beginners start OCaml programming "by themselves" (not in a controlled environment like a course with an educator giving precise instructions, but by random Google navigation as you describe). It is extremely, extremely frustrating -- OPAM is basically unusable for complete beginners for example, because of all the ways they manage to get outside the switch environment, which they don't really understand in the first place.

There is no good one-fits-all solution to this problem, we need massively more effort invested in the beginner experience and language tooling. Because this is a heck of a lot of work, I believe distributing is it the right move: we should all, collectively¹, strive to improve it bits by bits. For example, the people in charge of ocaml.org have invested a lot of work in being sure the website is dead simple to modify. If someone see something small that could be improved on this website, just do it!

¹: Granted, some parts of the ecosystem also require a more precise design vision and will not just materialize by distributed small efforts. I believe fixing the information surrounding the language can and must be done collectively.

All of that said, while I care dearly about encouraging people to contribute to the OCaml compiler distribution (and making this as easy as possible), and of course am glad of your efforts to improve the tooling ecosystem (the ocamldebug work for example).... I still believe that improving the ocaml toplevel is a waste of time in the medium-long term, because (1) utop exist as a better alternative and (2) there is no cost for users to switch to it (assuming a reasonable installation story, which I think is reasonable at least in the medium term).

@trefis, you can also use #utop_prompt_dummy;; for the same effect.

@diml Thanks that's very useful, it makes utop sufficently quiet for me to use. Might be worth to mention that somewhere in the documentation.

On the other hand, one could argue that there are various levels of improvements to the built-in toplevel that are not equal.

The level I'm thinking about (and I think was also @jwatzman's perspective in investing efforts in this pull request) is "let's make it a good toplevel everyone wants to use" (Best Toplevel on Earth). I think this direction is a waste of time, because utop is already miles ahead in terms of usability (and architecture allowing usability-features) and it is just as easy to contribute to it and improve it. Another level would be "if people stumble on ocaml by mistake or lack of information, let at least not be ashamed of ourselves" (Not a Shame). Regardless of our documentation effort, it is probably fair to assume that, as long as ocaml is installed, people will stumble on it. Efforts on a Not-a-Shame level would thus be justified¹ on a purely for-the-show point of view, and that would in particular suggest that, if this pull request can be made regression-free, it would be good to adopt it.

¹: and I would agree that having some history support, on systems where this does not require a large development effort (and this very PR seems to indicate that nowadays it doesn't), should be a basic requirement for any interactive toplevel.

(I think that having a disclaimer on interactive ocaml use to recommend utop either is the best of both worlds, because it provides a long-term usability solution to ocaml users at low development cost, and also encourages people to improve utop.)

@jwatzman , are you personally using utop? Are you colleagues using it? Besides visibility, is there any way you think it should be improved?

For example, the people in charge of the have invested a lot of work in being sure the website is dead simple to modify. If someone see something small that could be improved on this website, just do it!

Thank you for emphasizing this point. Please, just do it.

I (and probably the other ocaml.org team members) are well aware that the website needs a lot of work. The fact is we just can't keep up, and we need your help. Just resolving the open issues is hundreds of hours of work. I estimate that ocaml.org has already had about 3500 hours of work put into it (excluding opam.ocaml.org). Amusingly, when I first proposed the site, Xavier said something like "Well that would be nice, but it sounds like a $100,000 project", implying it wasn't feasible. Well, if you consider even the lowest hourly rate for programmers, we've now invested several times more than that. :)

I more than agree with your point above. [...]

Glad we're in agreement here. The change to ocaml.org is a nice start :)

@jwatzman , are you personally using utop? Are you colleagues using it? Besides visibility, is there any way you think it should be improved?

I actually am not; I'm not sure about my colleagues. The reasons for this are somewhat orthogonal to the issue at hand here (and could easily be their own rant :)). It boils down to being in a corporate build environment with certain opinions about the way packages should be installed and laid out, and opam/findlib having their own fairly inflexible assumptions about the ways packages should be installed and laid out which are incompatible -- all meaning that we can't use opam directly for our build env and so each package we want to build against, or otherwise include in the build env, has to be manually done one by one.

There's nothing stopping me from installing opam and utop locally for my own usage outside of our build env, but then it's up to me to keep versions in sync etc etc, and it just doesn't seem worth the compatibility headache. Some of my colleagues might do this, it wouldn't surprise me, but I'm not sure.

I still believe that improving the ocaml toplevel is a waste of time in the medium-long term, because (1) utop exist as a better alternative and (2) there is no cost for users to switch to it (assuming a reasonable installation story, which I think is reasonable at least in the medium term).

This would be awesome to see in the medium-long term. However, I think we both agree there's a lot of work to be done here before that happens, and this PR is nice and small and solves a very real usability problem in the short term. (i.e., it seems exceedingly unlikely utop will be the default for 4.03, and this PR is ready to go right now.) Can it get merged? :)

I think we would need someone familiar with OCaml I/O ( @xavierleroy , @damiendoligez ?) to confirm the fact that your C re-implementation of this OCaml function is correct.

Actually it doesn't need to be re-implemented, @jwatzman why not do this?

external libedit_available : unit -> bool = "..."

let read_interactive_input = ref (if libedit_available () then ...)

Also, a side effect of the current patch is that all native ocaml programs ends up with two new runtime dependencies: libedit and libtinfo. Not sure if that's a big deal but it's easy to avoid the same way it's done for the terminfo stuff: have a dummy implementation for native code and pass -ledit only for bytecode programs.

Also, a side effect of the current patch is that all native ocaml programs ends up with two new runtime dependencies: libedit and libtinfo. Not sure if that's a big deal but it's easy to avoid the same way it's done for the terminfo stuff: have a dummy implementation for native code and pass -ledit only for bytecode programs.

I think it's a big deal. Adding dependencies to native-code programs because of implementation details of the toplevel is quite distasteful.

We've had dynamic linking of C primitives for 14 years, so there's no reason to add primitves to the default runtime for the sole benefit of the toplevel (and indeed I'd like to see the terminfo stuff go away). It should be possible to move these C primitives to toplevel/, no?

Thanks for the responses everyone! Sorry to take so long to respond myself.

Actually it doesn't need to be re-implemented, @jwatzman why not do this?

I'm... not sure what I was thinking. I had it in my head that doing this extra external was bad for some reason, but it's certainly a lot cleaner than needlessly moving this code into C. I'll definitely do this.

I think it's a big deal. Adding dependencies to native-code programs because of implementation details of the toplevel is quite distasteful.

FWIW, libedit is considered even by the notoriously conservative Debian project to be the sort of thing that you should always have installed, so it's going to be available everywhere anyways. But I do understand your point that even given that, it's distasteful to include it everywhere since only the toplevel uses it.

One solution to this is to make the primitive available everywhere -- it's potentially quite useful in general -- but it is kind of a weird library to have available as part of the stdlib instead of, well, a library.

@diml suggested only doing it for bytecode programs, but @damiendoligez suggests just using the dynlink functionality, correct? (Those are two distinct solutions, to make sure I understand?) Skimming the current build system, it looks like a similar sort of thing might be done for bigarray, graphics, nums, str, and unix, I see cmxs files for all of those, are those good examples to look at -- both for dynlink and for how to plumb it through the existing build system? I will look into dynlink and revisit this PR in the next few weeks, when I get time.

@diml suggested only doing it for bytecode programs

Even if not compiled by default, there is also a native toplevel, ocamlnat, that should also benefit from such an addition.

Why not create an additionnal "otherlibs/edit", that would define both an OCaml module and its C stubs, and make "ocaml" depend on it ? Maybe the terminfo stuff could go there too ? Personally, I hate dynamic linking because it makes your program behavior depends on your environment in a complex way.

I wrote:

We've had dynamic linking of C primitives for 14 years,

In fact, the cygwin64 port contradicts me, as flexdll is not available there.

@damiendoligez : it just means that on cygwin64 we would be building the toplevel "ocaml" in custom bytecode mode. If we follow @lefessan 's suggested approach, ocamlmklib would take care of this all by itself.

@jwatzman : the problem is not Debian -- everything always works on Debian :-) The problem is Windows, Mac OS, and OCaml unikernels for Xen, where no / few libraries are available by default and you want to minimize ocamlrun's dependencies.

We discussed this PR at the development meeting last week. There seem to be consensus on the fact that:

• once this PR is ready, it should be merged
• there is actually a non-neglectible amount of work to be done to make it ready (sort out the (dyn)linking status, work on read_input_default again...)

Glad to hear it was even discussed, let alone there was consensus it should be merged :) Sorry I've been remiss in responding to comments here, and updating the PR. It actually could be into 2016 before I get back to this, unfortunately -- I've been super busy with a month and a half of international speaking, going straight into the US Thanksgiving holiday, and am now dealing with the mechanics of a transatlantic move, which will keep me busy until the Christmas holiday. I do absolutely intend to come back to this, maybe over Christmas if I get bored, or otherwise when I can slip it in sometime in early 2016 during a FB Hackathon :) If someone wants to take over this, let me know, otherwise that's my current timetable. Thanks so much for keeping on top of the reviews here!

@jwatzman Could I encourage you to finish this one off?

FWIW, I think this is a useful and sensible addition to the toplevel. In particular when working on the compiler or outside of OPAM for whatever reason, it may be difficult (and/or not worth the trouble) to install utop. Furthermore at least one common platform (macOS) as far as I know does not come with any rlwrap equivalent out of the box.

Much as I would like to see this feature merged, there hasn't been any work on it for a long time (close on two years)---and a reasonable amount of work is needed---so I'm going to close this pull request. If the original author or someone else wants to pick it up again, we can re-open it.

Yeah, closing this is sadly the right thing for now. I have changed projects at work and so no longer have quite the same incentive to hack on the OCaml compiler as I once did :) I'd love to pull this across the finish line, or see someone else do so, but it's unlikely I'll be able to any time soon, especially since the last round of review asked for a large portion of this to be reworked. (Probably rightly so! But that doesn't make it any less work :))

Sorry to have gone MIA for two years, but that's where I'm at.