when renamed as ocamldep, boot/ocamlc will behave as ocamldep

This is frowned upon by many, not least the GNU coding standards:
https://www.gnu.org/prep/standards/html_node/User-Interfaces.html

@xavierleroy Indeed, the current version of this PR would fail if somebody wants to rename ocamldep to my-system-prefix-ocamldep... I won't fix it unless there is some interest in it, in particular in sparing 2MB of space in the GIT repository at each bootstrap commit...

I support attempts at removing boot/ocamldep.

What about using the command-line to enable ocamldep behavior from withing ocamlc/ocamlopt? E.g. a sub-command syntax "ocamlc depend ..." (I think the risk of conflict with current use is tiny: all files passed on the command-line usually have an extension).

User code would need to be adapted, but one can always produce an ocamldep for some time and use ocamlc depend within the distribution itself.

ocamldep could become just a script that invokes the feature from the compiler...

This probably won't work under Windows, but is indeed an option elsewhere.

I think that a dedicated "sub-command" is better in this case than a regular option, because ocamldep itself has a number of command-line options than are irrevelant for the compiler.

Why not call the sub-command -depend or even -ocamldep instead of inventing a new syntax for command-line arguments?

@damiendoligez I don't understand your comment, which new syntax for command-line arguments ? The new version uses -depend as a subcommand, and parses the remaining arguments just as ocamldep would do.

@alainfrisch was talking about using depend (rather than -depend) as the subcommand name, and that is what I found objectionable.

Since you used -depend all is well.

The current Changes entry does not mention this PR #1078.
I think it should, to reflect the real developemnt history. @gasche ?

Regarding your changes to the Makefile: Since CAMLDEP does now call CAMLC,
I'd suggest to do as follows:

CAMLC=$(CAMLRUN) boot/ocamlc
CAMLCFLAGS=-g -nostdlib -I boot -use-prims byterun/primitives
CAMLDEP=$(CAMLC) -depend

And then replace the occurrences of $(CAMLC) by
$(CAMLC) $(CAMLCFLAGS)

@alainfrisch was talking about using depend (rather than -depend) as the subcommand name, and that is what I found objectionable.

I believe people are used to the notion of subcommand, and it would make it clearer that (i) the new option can only be used as the first argument; (ii) this option switches the CLI parsing to an entirely different set of command-line options.

Yes, assuming that a first command-line argument without an extension is a subcommand should work fine, since the files passed to the compiler always have an extension, as far as I know.

That holds for tools like git but not for compilers, in my opinion.

What's the difference?

I just wanted to emphasize that I personnally do not expect a compiler to take subcommands.

But what distinction do you make? ocamlfind, for instance, uses sub-commands to call compilers.

Adding a subcommand would make the interface a bit inconsistent, no ? git or opam always use subcommands. People would then expect to be able to also use "ocamlc opt" for native code, and so on.... Also, how to document this in ocamlc -help ?

Maybe I can just add the option, and we can later discuss a subcommand schema for OCaml ?

Maybe I can just add the option, and we can later discuss a subcommand schema for OCaml ?

Why not, but we would then need to maintain backward compatibility with -depend.

while as far as I know ocamlc and ocamlopt do not have a notion of subcommand so far.

Indeed. But we are talking here about including an "entirely" different tool, with its own command-line arguments, into an executable that does otherwise something else. This seems to be a good time to discuss this idea of using a sub-command syntax for such cases.

git or opam always use subcommands

Technically speaking, some features of git are accessible without a subcommand (git --help, git --version, git --man-path, etc).

Btw, did you consider also merging ocamllex into ocamlc?

Is this feature discussed for 4.05 or 4.06 ? I assumed it was for 4.06, so I would expect we would have time to decide on a subcommand scheme later.

We shouldn't restrict ourselves to what another command does, especially as old as gcc. If the goal is to decrease the size of boot/, then it would make sense to have subcommands for both ocamldep and ocamllex, that would probably only be used during the compilation of the distribution.

New version with the requested changes.

I also took this opportunity to add Depend.pp_deps, a variable that plugins can use to add dependencies (for example to a file that was included or used to generate some code). They will be printed in the standard output, but not in the raw output.

1. About the Changes entry: it says that the -depend option has been
   added to ocamlc. Hasn't it been added to ocamlopt, too?

Fixed !

2. About the Makefile. Your implementation indeed requires less changes
   than the one I have suggested (because it does not touch the CAMLC
   variable), but to be honest I find it a bit hackish.

I don't see why. The other solution would have uselessly added $(CAMLCFLAGS) to all the lines in the Makefile, whereas this one is much shorter. So, isn't it the other way around ?

About the help text:

Fixed !

To me this is not really related to this PR so I'd rather have this
submitted in a separate one.

I would have to create another git branch, another PR, start a new discussion, just for 5 lines of changes that do nothing if you don't use plugins. Is it really worth wasting that much time, just for the sake of respecting the "process" as much as we can ? I would do it if Damien tells me he wants to cherry-pick it to 4.05 :-)

They will be printed in the standard output, but not in the raw output.

I'm not sure I understand this sentence. Does this mean you output a different dependency graph depending on the textual format chosen for the output?

@damiendoligez It means that ocamldep -modules will not print these dependencies, as it should only print dependencies on modules.

It means that ocamldep -modules will not print these dependencies, as it should only print dependencies on modules.

This breaks the invariant that dependencies printed with "-modules" are a superset of the ones printed with ocamldep's default mode. At the very least, I am not sure to understand why Depends.pp_deps could not be used for module dependencies.

This breaks the invariant that dependencies printed with "-modules" are a superset of the ones printed with ocamldep's default mode

To the best of my knowledge, this subject has never been discussed before, I have never heard of such an invariant.

I am not sure why Depends.pp_deps could not be used for module dependencies.

Because the files that would be included in that list are not modules, so they have no .cmo, .cmi, .cmx ?

If there is some popular demand for it, I can add an option -ppdeps that would also print such dependencies, so that tools using ocamldep -modules can use this flag to print them in the future, but it would be foolish to print them currently, as OMake or ocp-build would wrongly interprete them.

@lefessan you need to update the man pages and the reference manual. I'll be particularly interested in seeing your specification of Depend.pp_deps...

@damiendoligez Currently, there is no change in the arguments of ocamldep for pp_deps, so I don't really see what to put in the manual for it. Do you mean that I should document, after adding also the -ppdeps option ?

I mean you should document the interface you offer to plug-ins, obviously.

You also need to document the -depend option of ocamlc and ocamlopt.

Yes, I should do it for -depend.

For the plugin API, where shall I start ? By describing all the internal data structures (parsetree, typedtree, lambda, clambda, cmm, mach) that are accessible to plugins ? Then, any modification in the code of the compiler would require a change in the documentation...

Honestly, I don't see what plugins add to the already existing compiler-libs, for which we did not provide any documentation. Ppx have been around too for a while now, and their internals (and API) are not documented either. For me, such APIs are not to be used by OCaml developers, they are supposed to use existing plugins, not to develop them. If we start going into another direction, we should also reject any modification of the compiler that is not commented enough for any OCaml developer to understand it, as he might use that code through plugins or compiler-libs...

Honestly, I don't see what plugins add to the already existing compiler-libs, for which we did not provide any documentation.

Hooks are entry points to be used by external code, by definition. Ideally all internal APIs should be documented (and some parts are indeed being incrementally documented), but new entry points deserves a few lines of documentations. It's not about specifying formally all internal data structures and invariants, just adding a few lines of docstrings for each hook. Explaining what the Depends.pp_deps is and how it is supposed to be used could be done in probably 5 lines of texts and would make it much more useful.

Ppx have been around too for a while now, and their internals (and API) are not documented either.

What do you mean?

I find Ast_mapper and Parsetree rather well documented, and it is roughly the API surface required for ppx.

For me, such APIs are not to be used by OCaml developers, they are supposed to use existing plugins, not to develop them.

Hooks are added so that they can be used. Hopefully not by newcomers, but not necessarily only by core developers (and even for them, a few lines of explanation would be useful).

Ppx have been around too for a while now, and their internals (and API) are not documented either.
What do you mean?
I find Ast_mapper and Parsetree rather well documented, and it is roughly the API surface required for ppx.

I just discovered chapter 24 of the manual, I was previously mistaken by the missing link in the documentation of -ppx here:
https://caml.inria.fr/pub/docs/manual-ocaml/comp.html

Explaining what the Depends.pp_deps is and how it is supposed to be used could be done in probably 5 lines of texts and would make it much more useful.

I will do it once there is an agreement that the PR will be accepted.

the missing link in the documentation of -ppx here: https://caml.inria.fr/pub/docs/manual-ocaml/comp.html

Thanks for pointing the missing link, see #1129 for a fix.

If there is some popular demand for it, I can add an option -ppdeps that would also print such dependencies, so that tools using ocamldep -modules can use this flag to print them in the future, but it would be foolish to print them currently, as OMake or ocp-build would wrongly interprete them.

Another possibility would be a flag (-ppdeps or -only-ppdeps?) for printing only dependencies added through pp_deps. Anyway I would argue that such an option is necessary, otherwise external tools (for instance ocamlbuild) will have to parse the generated makefile to retrieve these dependencies — which does not sound optimal in term of usability.

I updated this PR:

• It is not used anymore in the current Makefiles. The idea is to first accept the PR, wait for a bootstrap of the compiler, making -depend available for the Makefiles, and then provide a new PR to use it and remove boot/ocamldep
• the manual and the man pages have been updated with a short description of -depend

@damiendoligez Do you need more changes for acceptation ?

@shindere I think the header date is OK, and the file name doesn't look so bad. We can change it later if needed.