Conversation
| File "libc/c3.ml", line 1, characters 8-11: | ||
| 1 | let x = A.x + 1 | ||
| ^^^ | ||
| Error: Unbound module A |
There was a problem hiding this comment.
Here's a suggestion for an improved error message:
Error: The module A is unbound because it is hidden. (See manual section ....)
Or
Error: The module A cannot be referred to directly, because it is only found in a hidden (-H) include path.
(See manual section ...)
We would need to propagate how a particular path was found, but it seems like that should be straightforward.
|
I've pushed two additional commits. The first slightly reworks the file cache in The second commit fixes the typo in the changes file and gives the test reference files more descriptive names. I'm hesitating to make the suggested change to the error message, because I'm not sure it's actually helpful to users in the typical use case. To say more: The common scenario I'm imagining is that users will write out explicit dependencies for the build system (say in a dune file), and the build system will pass those to the compiler with -I and then compute all the transitive dependencies and pass them with -H. If the program directly refers to something that's only available through -H, the appropriate remedy is for the programmer go to list that module in their dune file. In the current world, they don't ever need to know about the existence of -H - they get the same error as for any module that's not on the load path, and apply the same fix. But with the revised error, they'd have to understand -H and what their build tool is doing about transitive dependencies. |
|
It is true that most users will interacts with the
In that case one can fix with: But get the error Having a different error message would help to directly remember the problem and the fix, moreover the build system can match this error message and add its own hint. |
lpw25
left a comment
There was a problem hiding this comment.
A couple of small comments. Looks good to me. Simpler than I was expecting, sorry for not reviewing it sooner.
| let mk_H f = | ||
| "-H", Arg.String f, | ||
| "<dir> Add <dir> to the list of \"hidden\" include directories\n\ | ||
| \ (Like -I, but the program can not directly reference these dependencies)" |
There was a problem hiding this comment.
I think is is differently indented by one space compared to the similar entries in this file. Presumably that might make the message indentation off by one too.
There was a problem hiding this comment.
The indentation of multiple-line descriptions is a bit inconsistent in this file, but I count six flags where this indentation (5 spaces) is used for subsequent lines: -opaque, -strict-formats, -no-strict-formats, -for-pack, -w, -warn-error. This makes it the most common choice. I checked just now, and it prints like them.
There was a problem hiding this comment.
Note: the absence of printing difference comes from the combination of escaped newline and escaped space:
The following two string literals have the same value:
let s1 = "foo\n\
\ bar"
let s2 = "foo\n\
\ bar"
|
I have squashed this PR to one commit, since it no longer had a clean history, and rebased it. There were a few conflicts related to #12389, but they were easily resolved. I think the only remaining question here is about whether a new error should be added for the special case that the program mentions a module that would be available if they changed a -H flag to a -I flag. I continue to think the current error is best, since most users won't know about -H at all, and the fix is the same as for all other "Unbound module" errors. Regrading @bobot's most recent remark that "users could be surprised that they can't mention a type that is present in error messages" - this is already the case before this PR (see the missing cmi tests). I think this is fine to merge, once CI passes, if others agree. We can always add a new error later if it turns out this is source of confusion in practice. |
There was a problem hiding this comment.
Let me add my voice of support for this PR. It is a notable new feature that I care about (in particular: it is the first significant compiler-level improvement proposed by the Dune people), and the new implementation, following the design recommended by @lpw25, is clearly simpler and better than the one @bobot and myself attempted in #11989. I am in favor of merging.
(I am "approving" to mark support but I only skimmed through the implementation and have not reviewed it in full. I would not consider myself a reviewer, but I understand enough to assess that I am confident in your work and Leo's review.)
Minor points/comments:
-
I was expecting (secretly hoping?) that there would be a comment somewhere in the compiler codebase that gives a high-level picture of how modules are looked up in the load path / initial environment, to explain the interleaving of -I, Stdlib, -open arguments etc. (I remember asking for a precise description of this when Jérémie Dimino was working on -open.) If this existed, then the present PR should update it. I have been unable to locate it again in the compiler codebase. There is a comment in compmisc.ml that you updated as expected, but there is also a comment at the beginning of load_path.mli that is getting stale (it also lacks a mention of the
-cmi-fileoption I think), could you update it? -
In #11989 we wondered about an example where one accesses a "hidden" module through constructor disambiguation, without having to name the hidden module directly.
(* A.ml : imported hidden *) type t = Toto (* B.mli: imported visible *) val f : A.t -> unit (* C.ml: being type-checked *) let () = B.f Toto
I was of the opinion that this example should be rejected, and François that it could be accepted. Can you confirm what the behavior of your implementation is on that example? (My guess would be that it is rejected, but I am not sure.) I am not saying that either behavior is clearly wrong and would be ground to change the code, I think that in this case it is better to "let the code speak" and accept the behavior given by the simplest implementation. But we should be explicit in the discussion about how those edge cases are resolved.
-
What are the plans for the
-cmi-fileoption, which did not exist when the RFC was originally written? I would expect a-hidden-cmi-filevariant. I am not saying that this must be integrated in the present PR, but I would like to see it implemented if there is agreement that it also deserves a hidden variant. What do we think? Are you volunteering to implement it in a followup PR?
manual/src/cmds/ocamldoc.etex
Outdated
|
|
||
| \item["-H" \var{directory}] | ||
| Like "-I", but the "-H" directories are searched last and the program may | ||
| not directly refer to the modules included this way. |
There was a problem hiding this comment.
The wording here is not great, but nobody cares because these are just ocamldoc options that duplicate compiler options. The current approach (adding meh formulations to a manpage no one reads anymore) is acceptable, but in the medium term it would be nice to get rid of these redundant option descriptions. ( @Octachron may know what would be a good way to do this.)
There was a problem hiding this comment.
My general plan for this issue would be to have a unique source of truth for all flags and their documentation and generate all formats and slices from this unique input.
I would be in favor of doing this if possible. It looks like a very simple change, changing By the way, can we bikeshed on
I would definitely want to have this in the toplevel. I regularly use the toplevel to test the type-checker, and any mismatch between compilation options and toplevel options is going to block me down the road. |
|
@gasche Thanks for the feedback.
I will have a go at |
|
Thanks! Maybe it would be useful to have this type-directed-disambiguation corner case in the testsuite. |
|
I found a few more issues, while working through some of the suggested changes, so I'm pushing 5 more commits - the 4th is significant and needs review in particular. The commits are:
The bug fixed in the fourth commit is that, once a -H module had been read in for some legitimate type-checking purpose, it was added to the To fix this, the Finally, with respect to the toplevel: It turns out I was wrong, and |
|
The toplevel behavior you describe is entirely reasonable, and not dealing with extra directives is fine. @lpw25 would you by chance be available to do another round of review specifically on the new issues that came up and the changes to the implementation? |
lpw25
left a comment
There was a problem hiding this comment.
One bug that needs fixing and one opportunity for simplification.
| | Found (ps, pm) when allow_hidden || ps.ps_visibility = Load_path.Visible -> | ||
| (ps, pm) | ||
| | Found _ -> raise Not_found | ||
| | Missing -> raise Not_found |
There was a problem hiding this comment.
I think the logic around Missing isn't quite right. If you first try to load a file with allow_hidden is false then you'll set it to Missing in the cache, and that will prevent future lookups even when allow_hidden is true.
There was a problem hiding this comment.
For future reference: Leo and I discussed this offline. It should be difficult to get bad behavior out of this, because generally if ~allow_hidden is false we're checking an explicit reference to something in a program, and a failure here will result in a failure of typechecking.
We're not confident that's always the case, though, so I'm pushing a fix which is to only add Missing to the cache if allow_hidden.
An alternative would be to add an argument to Missing to track which kind of lookup has failed. That would have the advantage of avoiding multiple calls to can_load_cmis, but only in the rare case that typechecking succeeds after multiple failing calls to find_pers_struct for the same module with allow_hidden=false. (And the disadvantage of marginal additional complexity and allocation).
There was a problem hiding this comment.
The only case that I can think of as potentially problematic would be in Merlin, if they cache failed lookups usefully. (But this assume they watch the filesystem somehow to know when to clean the cache.) Do we know if they do that? Probably not?
(I would be slightly more at ease with caching both with an extra argument to Missing, which we are sure will have no adverse side-effect.)
There was a problem hiding this comment.
IIRC they already have to patch this stuff to get their desired behaviour anyway
utils/load_path.mli
Outdated
| type visibility = Visible | Hidden | ||
|
|
||
| val find_normalized_with_visibility : string -> string * visibility | ||
| (** Same as [find_normalized], but also whether the cmi was found in a -I |
There was a problem hiding this comment.
"but also whether [..]": but also reports/returns/indicates?
|
There is a Windows test failure in the testsuite that I am not sure how to address: ... testing 'test.ml' with 1.1.1.1.8.1.1 (check-ocamlc.byte-output) => failed
(compiler output tests/hidden_includes\test\ocamlc.byte\ocamlc.byte.output differs from reference tests\hidden_includes/wrong_include_order.ocamlc.reference:
@@ -1,3 +1,3 @@
File "libc/c1.ml", line 1:
-Error: The files "libb/b.cmi" and "liba_alt/a.cmi" make inconsistent assumptions
+Error: The files "libb\b.cmi" and "liba_alt\a.cmi" make inconsistent assumptions
over interface "A"
)(several tests fail for the same reason) I would be tempted to disable those tests under Windows (just add |
|
@gasche Thanks. I've disabled the relevant tests on windows (I think) and fixed the typo you spotted. I won't have time today to rework |
|
Re. Missing: I have a preference for the cautious route that always caches failures, with a new argument. (I think that this is a more systematic approach for memoization of this function anyway, so the code is simpler / more regular to think about.) On the other hand, I am tired of pulling your leg on this PR. Let me merge know, and you can improve |
|
It looks this change unwittingly made things harder for ppxlib maintainers, see ocaml-ppx/ppxlib#451 (comment) and then ocaml-ppx/ppxlib#475 . ppxlib is affected by a format change in the ocaml.ppx.context AST payload that comes from 01178bb#diff-e4e6b5f588112250b3281c08994382dce4893de604562364d84737a14d3bec43 . Because this change is only in 5.2 and not an older release, we have an opportunity to consider doing things differently to make ppxlib's life easier. I know little about this ppx.context things, but here are options that come to mind:
I suspect that the "support both formats at once" approach may make things more complex, and cause trouble if the ppx contest of a recent release is consumed by something linking against an older release. So from a distance (1) sounds easier, but I know little about these things. I am also not completely sure of whether this issue is still relevant, or whether the ppxlib people found a change that they like to support arbitrary format differences here. |
|
For full compatibility, it might be better to output At the same time, the feedback that I got from @pitag-ha and @NathanReb yesterday was that handling the change in ppxlib worked well enough after adding a compatibility layer. However, we might want to document an update policy for the ppx context? |
|
We discuss this today, there seems to be consensus for @Octachron's proposal (keep the field unchanged, add two new fields that are more precise) if we want to preserve backwards-compatibility. We don't really understand what the ppxlib people prefer at this point (and if it's for good reasons or just to be polite to us), so he will ask again. |
|
We have just implemented a migration for the new It's probably worth mentioning that this change only caused trouble in a very specific case, that is: running a ppxlib driver compiled with 5.2 (or higher) on an older marshalled AST. In short, there was no strong enough incentive on our side to push for @Octachron's above proposal but a non breaking approach would indeed be our preferred way as we do rely on compiler-libs to read |
|
Thanks a lot, everyone! @Octachron and @NathanReb have already explained most of it from the For us, it is of course always nicer if everything we use from the compiler - including the PPX context - can be kept stable: The less migrations on our side, the better (less code complexity, less room for bugs or compatibility problems). We haven't hit you up on this, because we're aware that we're using the
Yes, it's unclear to us if there're still users with a set-up (i.e. compile the PPX driver with a different compiler than the project) under which this change would break the PPX driver. Everyone who uses In short: If you can make the ppx context change non-breaking without much effort (e.g. in the way you've discussed), that's perfect for us. However, if it'd be much effort for you, it's no problem for us if you keep it as is: We can introduce the ppx context migration into |
|
I'm sorry that this PR caused issues for ppxlib maintainers! I did update our internally vendored copy of ppxlib to support I am only minimally familiar with ppxlib, but my first reaction is that this PR has exposed a pre-existing issue - ppxlib is relying on some compiler-libs functionality not to change, but there is no guarantee of that. My naive instinct is that rather than trying to avoid a breaking change in compiler-libs, we should introduce some explicit versioning into the datastructures ppxlib is reading, so that ppxlib can have a nice migration story that plans for future changes (not that I'm suggesting to start introducing more changes, and in retrospect I clearly should have communicated with ppxlib maintainers about this PR earlier). Of course, I am not an expert on ppxlib or the versioning policy for compiler-libs, have not taken the time to read the proposed ppxlib patch in detail, and am proposing to create a bunch of work for ppxlib maintainers. So I can easily be convinced this view is incorrect. Either way, I'm happy to help to the extent I can, once I'm back at work next week. |
|
Just for info, Dune has added support for this flag: ocaml/dune#10644. |
Does it support the |
|
It seems orthogonal. It is the re_export feature of dune, that would be linked to your META |
|
Well I think in terms of user experience not features… I think |
To be clear, the As @bobot mentions, it is likely there is a partial overlap between the semantics of this field and the |
|
Ok thanks @nojb. Peculiar that no dune dev mentioned the existence of this field in that discussion when we were trying to find a good name. In any case with the Unicode release around the corner I will soon start shipping |
|
For future reference, this discussion ( |
This PR is a reimplementation of #11989, which addresses RFC 31, following a suggestion of @lpw25's to avoid an issue with the previous approach.
It adds a
-Hflag, which is like-Iwith two exceptions:-Hdependencies go at the very end of the search path.The previous attempt allowed mixing the
-Iand-Hflags and preserved that order in the search path. This meant, to handle shadowing, some-Hmodules needed to be added to the typing environment, which had to be updated to track which modules were included in which way. And this potentially creates reproducibility issues along the lines of those fixed in #9991. Leo's suggestion is to instead put the-Hmodules at the end of the search path, where they can't shadow anything.I think the result is a win. There is a bit more work in
load_path.{ml,mli}to keep track of the-Iand-Hdirectories separately, but fewer changes in the type system, and the shadowing problem is avoided.I've revised and expanded slightly on the testsuite from #11989. I updated
ocamldocandocamldepto support the new flag, and added it to the manual.A few things I haven't done:
cmtformat to distinguish the-Iand-Hdirectories. It's not clear to me whether this is needed - I took a quick look at merlin, but it doesn't seem to use this part of the cmt.-H. I don't think either is hard to do, if it would be useful.I have listed @bobot and @gasche as co-authors in the Changes entry as this takes a little code from and is inspired by their attempt, but any mistakes here are mine.
@lpw25 has agreed to review this, though of course feedback from others is welcome. I've broken it into a few commits - the first updates the core system, the second updates the debugger and ocamldoc for compatibility, and the third updates the manual. Edit: but note the changes to the file cache in load_path have been tweaked for efficiency in another commit made after this was posted.