This seems awesome ! Please write a short description for everybody to understand what it is without looking at the changes :)

Here is a description of the changes, recovered from Alain's clear commit messages:

• This PR imports an updated version of the local_exn.diff patch
  submitted on http://caml.inria.fr/mantis/view.php?id=5879

• It adds local exception declarations to expressions:

   let exception C of ... in
   ...
  

  (This introduces a conflict in the grammer, to be investigated.)

• Compile uses of such local exceptions to staticcatch/staticraise when
  the constructor is used only locally as an exception (i.e. raised or
  caught) in the current function block. The optimization is done
  as a rewriting in the Lambda code, which requires to reverse-engineer
  the code compiled for the declaration, the raise, and the catch.
  Alternatively, one could trigger the static compilation scheme
  in translcore while compiling the local exception expression.

• a [@static] attribute can be used (let exception[@static] ... in ...) to get a warning when the static compilation scheme cannot be
  used.

let exception[@static] C of int in ...
let exception C of int [@static] in ...

Do you really have two different syntaxes for the same thing? By analogy with let open, only the first one should be accepted.

Other than that, I like the feature but I'm a bit worried by the "reverse engineering" part of the implementation.

Other than that, I like the feature but I'm a bit worried by the "reverse engineering" part of the implementation.

This is indeed not very satisfactory. Detecting that a local exception can be compiled into a static one during the translation from Typedtree to Lambda would seem preferable, but this is not straightforward either, in particular if one wants to support try..with block with both static and non-static exceptions (perhaps combined with or-patterns).

If I understand well the purpose of this patch, we should introduce a new construct in the language, that does not make it more expressive, but just allows the compiler to optimize the static case. I don't think it is a good idea to extend the core language for such purposes. That's the purpose of annotations, while the core language should remain simple and clean.

For your case, I would argue to add the annotation on existing exception definitions:

let f x = 
  let module M = struct exception[@static] C of int end in
  try
   ... raise (M.C n)
  with M.C m -> m

It is more verbose, but at the same time, such optimizations should remain rare in standard code.

Note that, from what I understand, try List.iter (fun x -> if ... then raise (c x)) with c x ->... will not be optimized as static, making the whole use-case not very frequent in functional code.

1. We already have "let open" and "let module"; "let exception" seems natural enough and useful to express that the exception remains local (in particular, it cannot be matched by patterns which are not in the local scope); moreover it avoids polluting the global environment, and this is now even more important since exception names must remain unique in a given structure. Another minor advantage over encoding with "let module" is that even without the static compilation scheme, this is lighter at runtime as it avoids one allocation. So I argue that "let exception" is a light and useful addition, which could be defended in itself. (Btw, "let type" would also be useful to define local record/sum types.)
2. The optimization is not triggered by the attribute. The attribute only allows to be notified when the optimization doesn't apply; the optimization itself is tried automatically. An explicit local version of exception definition is a much more natural candidate for triggering the optimization than detecting a specific form of "let module".

We already have "let open" and "let module"; "let exception" seems natural enough and useful to express that the exception remains local

Fully agree. We should really have let versions of all the structure items.

We already have "let open" and "let module"; "let exception" seems natural enough and useful
to express that the exception remains local (in particular, it cannot be matched by patterns
which are not in the local scope);

let open and let module are not just syntactic sugar, whereas "let exception" is.

moreover it avoids polluting the global environment, and this is now even more important
since exception names must remain unique in a given structure.

but still, "let module M = struct exception E end in ... " does not pollute the global env either.

Another minor advantage over encoding with "let module" is that even without the static
compilation scheme, this is lighter at runtime as it avoids one allocation.

It would be worth checking that this one allocation is not removed by flambda, as the module itself is probably never used. Anyway, my whole argument is that we should not ruine the language just to be a bit faster. The whole point of an optimizing compiler is to avoid that, by optimizing such patterns.

So I argue that "let exception" is a light and useful addition, which could be defended in itself.
(Btw, "let type" would also be useful to define local record/sum types.)

I am sure we can be very creative to make OCaml programs unreadable and unmaintainable.

I strictly prefer

let f x =
  let module M = struct
     type t = ...
     exception M of t
  end in ...

to

let f x =
  let type t = ... in
  let exception M of t in
  ...

because code from the first one can be copied directly to toplevel, whereas code from the second one needs to be changed to be moved upwards.

The optimization is not triggered by the attribute. The attribute only allows to be notified when the
optimization doesn't apply; the optimization itself is tried automatically. An explicit local version
of exception definition is a much more natural candidate for triggering the optimization than
detecting a specific form of "let module".

I would actually expect the contrary: the optimization should be done today on exceptions defined in local modules (which can already be defined), and maybe later in local exceptions if they are accepted in the core language. Submitting both together is misleading, as people might want the optimization and thus support the PR, and then are forced to accept local exceptions that they don't care about.

Submitting both together is misleading, as people might want the optimization and thus support the PR, and then are forced to accept local exceptions that they don't care about.

I cannot imagine that someone wanting to use the optimization would prefer to use "let module M = struct exception ... end in" over "let exception ... in".

let open and let module are not just syntactic sugar, whereas "let exception" is.

No, let exception is not just syntactic sugar, and cannot be treated as such. Try creating, say, a camlp4 extension to support it, and you'll only be able to create a fragile approximation it. Consider:

  let exception E in ...
  let open M in ...
  ... raise E ...

There is no way to now purely syntactically that the raised E is indeed the local exception (M could redefine another exception E).

let open could also be treated "almost" as syntactic sugar, but not completely. (FWIW, the encoding used by my very old camlp4 openin syntax extension was: let open M in e--> let module BLABLABLA = struct open M let r = e end in BLABLABLA.r.)

I cannot imagine that someone wanting to use the optimization would prefer to use
"let module M = struct exception ... end in" over "let exception ... in".

I do. Actually, I even had such a patch, that optimized only that pattern.

And I hate when you have to explain to beginners that their "let module M = struct exception E end in ..." was not optimized because they didn't know that there was another construct "let exception E in..." that was the only one that was optimized, and is just in the language for that reason.

I think we should stop extending OCaml "because it is possible", and extend it only "when it is needed".

@lefessan does your patch also optimize

let f x =
  let module M = struct
    exception A of a
    exception B of b
  end in
 ....

If I need several local exceptions, I would undoubtely write this and possibly get the unoptimized behavior. At least local exceptions remove that temptation.

Being forced to invent two names (M/E) just for creating a forward jump label is just ugly (being forced to declare the jump is already quite heavy, but I'm ready to compromise on it). Yes, it is not "needed" to introduce local exceptions, but I maintain that it is a natural addition to the language. Once introduced, nobody would think about writing let module M = struct exception E end in ... so don't worry, you will not have to explain why that one is not being optimized (at least anyone aware of the optimization will know about local exceptions). On my side, I'd hate having to explain that the only way to get a cheap forward jump is to write let module M = struct exception E end in ... raise M.E ...

I completely agree with Alain here. Many times I've needed something like a local exception, and instead had to add a global exception. I never even thought about wrapping it in a local module, which is the problem -- thinking in terms of local modules is not simple or very natural IMO.

And I hate when you have to explain to beginners that their "let module M = struct exception E end in ..." was not optimized because they didn't know that there was another construct "let exception E in..." that was the only one that was optimized, and is just in the language for that reason.

I agree with this, it is better to use a more general analysis to detect whether an exception can escape rather than only optimizing a specific syntax. However, at the moment quite a few similar optimizations are annoyingly tied to syntax and until we improve the mechanisms in the middle-end for performing such analyses that is probably not going to change.

I think we should stop extending OCaml "because it is possible", and extend it only "when it is needed".

Adding support for let definitions of all structure items simplifies the language: users don't have to remember the quite arbitrary restriction that only open and module have local versions.

Is it much more difficult to implement?

It depends exactly what you have in mind. Would the local module contain only one exception? Several of them? Also other components?

In any case, it would add some complexity to the "reverse engineering" part, which is already quite intricate. More importantly, I don't see any compelling reason to do that: anyone aware of the feature would naturally use the local exception form. It should be quite straightforward to know in advance when the optimization would be triggered (as for tail-calls), so making its detection more and more clever for no direct benefit doesn't seem a good direction to me.

I would like to understand the reverse-engineering part better then -- whether it could be avoided by asking Luc to do part of the work in the pattern-matching part, and why it is problematic to support `let module).

My cursory understanding is that you are reverse-engineering the case-tree produced by the pattern-matching compiler from the try .. with .. construction. So the difficulty is more in the try .. with part than the let .. exception part. Is my understanding correct that, with the current code, the let exception declaration has to come just before the try .. with .. doing the raises? So you would have to relax that restriction, by handling any let-binding that may declare static exceptions (more than just let exception) and support the fact that there may be some statement (to be checked by the no-escape condition) between this binding and the corresponding try..with block? (Typically a let open in but maybe arbitrary code). This would also support the use-case where there are several try..with blocks using the same static exception, wouldn't that also be nice?

Wouldn't it be easier to do this analysis and rewriting at the typedtree level, so you don't have to do any reverse-engineering?

P.S.: the reason I'm wondering about this is that I would like to have the property that let <structure item> in <expr> and let Fresh_M = struct <structure item> end in open Fresh_M in <expr> are equivalent. This matters in practice: consistency is one aspect of good language design.

The current approach does not impose that let exception is just next to the try...with.... There can be multiple try...with... blocks in the scope of the let exception, and they can even be in an inner function.

Doing the work during the translation from Typedtree to Lambda is the other option, but it requires indeed a stronger interaction with the pattern matching compiler. My guess is that it would make the implementation more complex, although perhaps more robust. And conceptually, optimizations are typically done in later phases than this Typedtree->Lambda translation, which allows to better interact with other optimization (which could argue to do that later than Lambda, but I'm interested to have static exceptions available for js_of_ocaml).

I was thinking of doing a transformation pass on the typedtree, instead of having to interact with the pattern-matching compiler. I agree that Lambda (or later) seems a more natural target for optimizations, but we would need an escape analysis approach that is more robust that the current reverse-engineering.

Note: flambda should have some framework in place for non-escaping allocations, would it make sense to do the optimization there? Re. js_of_ocaml, @hnrgrgr worked on delaying the bytecode generation pass to after flambda-generation, and while this is not in discussion anymore for now, I think this should be kept in mind as an indication that flambda could eventually be appropriate even for js_of_ocaml scenarios.

I was thinking of doing a transformation pass on the typedtree

This is going to be tricky if you want to preserve sharing of handlers for static and non-static exceptions:

 let exception E in
 try ...
 with E | Exit -> ...

I think this should be kept in mind as an indication that flambda could eventually be appropriate even for js_of_ocaml scenarios.

This is good to know, thanks.

This is going to be tricky if you want to preserve sharing of handlers for static and non-static exceptions.

You could first rewrite this into try (try ... with Exit -> raise E) with raise E -> ... before doing your magic. (But another issue with working at the typedtree level is having to preserve type information.)

Indeed, it might be worth trying something along these lines.

Cases such as

let exception E of t in
...
try ...
with Exit | E _ -> ...

should be treated with care (inventing an argument-less local exception?).

In fact I don't think such a rewrite would be necessary, if you try by doing it at the typedtree level by lifting static{raise,catch} to the typedtree (I think you had patches around doing just that). In this case you can just recover sharing by having both cases static-raise to a shared handler, or do whichever clever thing your current implementation is doing, at the typedtree level.

It would be quite weird to include in the Typedtree representation some constructions not available in the surface language, just to be able to express some optimizations as Typedtree->Typedtree transformations. The Typedtree is supposed to keep the same structure as the Parsetree (annotated with type information). People already complain that optimizations are too tightly coupled to the concrete syntax...

On the contrary, I find that expressing interesting optimizations as source-to-source optimisations (whenever possible) is quite elegant. But you obviously have more experience on OCaml optimisations so I'd trust your gut feelings. Then I'd suggest to maybe look at the flambda level if it can avoid some of the reverse-engineering work (by using more general analyses already implemented).

On the contrary, I find that expressing interesting optimizations as source-to-source optimisations (whenever possible) is quite elegant.

I agree! But the logical consequence would be to extend the surface language with a dedicated construct for static raises (even if it does not allow to mix static and non-static exceptions in a or-pattern), which was my first choice.

Another direction to simplify the reverse engineering part would be to be more restrictive in the optimization, not supporting mixed try...with combining static and non-static exceptions (in the same clause or different ones); and perhaps annotating trywith nodes in the lambda code with the information of whether they include a catch-all case (this must disable the optimization).

It would be quite weird to include in the Typedtree representation some constructions not available in the surface language

It would be more than weird. How would you implement Untypeast.untype_structure?

I like the feature, as @damiendoligez , the implementation is not very satisfied. Actually, I like the original proposal on mantis(I think its implementation is also more elegant compared with this one and more lightweight). For the javascript backend, the local exception is not just an optimization, it's a feature as a control flow(the real exception in Javascript is terribly slow)

Now that local exceptions are present as a language feature, are there any plans to add the optimization?