Additional motivation for this patch: without this function, one cannot define bprintf and formatter_of_buffer in "user land". I don't see any reason to prevent that (and it can be useful for special purpose formatters).

Very nice, thanks.

Merged in trunk ( 33cf3f1 ), thanks!

@Drup I do not agree with your arguments:

• bprintf is error prone and this is a bad idea to resurrect it (use the hint given in the Format documentation).
• formatter_of_buffer does not use flushing (there is no sense to try to flush a Buffer.t!)

@seliopou I cannot grasp your motivation here: you never need to flush the formatter when using ikprintf or kfprintf: the continuation gets the formatter with all the necessary material; hence, you just have to go on printing on the formatter and everything will go smoothly! On the contrary, flushing the queue will just ensure that you do not want to interact with the preceding printing material and boxes structure.

To summarize: flushing the pretty printer queue was left private to the module on purpose. It would be confusing to use this facility, since it does not interact properly with the pretty-printing strategy (all of a sudden all currently opened boxes are closed and everything is printed instead of letting the pretty-printer do its job). The print_flush facility was simply added for interactive usage: when asking a question to the user you absolutely need the entire question to be displayed on the output device of the pretty-printer, hence the automatic flushing of the underlying output device.

@pierreweis I believe you are making assumptions about the underlying output stream attached to the formatter, in addition to assumptions about usage patterns (i.e, "go on printing"). Without this API, custom formatters that have not yet flushed their queues and are guaranteed to never receive further input will never flush their queues, except for perhaps at the end of the program. In a server application, programs don't terminate at the end of a request, so there will be no way of receiving formatter's output.

I cannot speak to the justification for this operation previously being private, nor reconcile that with why print_flush was public. The point of this patch is to adapt custom formatters to use-cases where it is known that no further input will be received and waiting for the program to terminate to flush the formatter's internal queue is not acceptable. I believe it accomplishes that goal. Whether that's a use-case that the OCaml maintainers want to support is another question, but the use-case is not without motivation.

Thank you for your answer.

@pierreweis I believe you are making assumptions about the underlying
output stream attached to the formatter, in addition to assumptions about
usage patterns (i.e, "go on printing"). Without this API, custom formatters
that have not yet flushed their queues and are guaranteed to never receive
further input will never flush their queues, except for perhaps at the
end of the program. In a server application, programs don't terminate at
the end of a request, so there will be no way of receiving formatter's
output.

Lot of questions: if you use kfprintf, it means you want the get the
formatter in your continuation, then is'nt it to go on printing ?
What assumption am I supposed to do about the output stream attached to the
formatter ? I was not even assuming the formatter being attached to a stream!

I'm sorry but I still do not understand your use case: could you define what you
mean by a ``custom'' formatter ?

In any case, if you are using the new function to flush the queue then the
``custom'' formatters you mentioned may never flush their underlying output
device and you will loose their output as well, no ?

Flushing the queue without flushing the device is just a source of eroor in
your ``use-case'', no ?

I cannot speak to the justification for this operation previously being
private, nor reconcile that with why print_flush was public. The point of
this patch is to adapt custom formatters to use-cases where it is known
that no further input will be received and waiting for the program to
terminate to flush the formatter's internal queue is not acceptable. I
believe it accomplishes that goal. Whether that's a use-case that the OCaml
maintainers want to support is another question, but the use-case is not
without motivation.

I always used to support user's concerns as much as possible. I'm still doing
it here since I'm discussing your point in depth. However, I do not want to
introduce a feature that will be error prone for other users!

So please, let me understand why this error prone functionality is in fact
needed before introducing it...

Regards,

Pierre Weis

INRIA Paris, http://bat8.inria.fr/~weis/

It appears that I merged the PR a bit too soon, as @pierreweis, the maintainer of the Format module, has objections. I'm thinking of reverting the merge and letting you ( @pierreweis , @seliopou , @Drup ) come to a consensus.

Lot of questions: if you use kfprintf, it means you want the get the
formatter in your continuation, then is'nt it to go on printing ?

The use of kprintf is so that the types works out within a library that I'm writing, but the need is not specific to kprintf. I simply discovered the limitation of the module through that function. The same situation may arise through multiple calls to fprintf. In my particular use case, there is only one call to kprintf, after which the internals of the custom formatter are collected and returned to the user.

What assumption am I supposed to do about the output stream attached to the
formatter ? I was not even assuming the formatter being attached to a stream!

I'm sorry but I still do not understand your use case: could you define what you
mean by a ``custom'' formatter ?

A custom formatter is the result of a call to Format.make_formatter. Here's a block of code that illustrates the use case:

let fprintf format =
  let tref     = ref noop in
  let formatter =
    Format.make_formatter
      (fun str pos len -> tref := !tref ^^ string ~pos ~len str)
      (fun ()          -> tref := !tref ^^ flush)
  in
  let k fmt =
    Format.pp_flush_formatter fmt ();
    !tref
  in
  Format.kfprintf k formatter format

The format string is used as an edsl, producing a data structure that represents the writes and flushes that must be performed. The result can be composed with other write and flushed operations (some of which are defined via generator functions). Additional library code can then traverse the representation, scheduling writes and flushes as defined by the user. Without a call to Format.pp_flush_formatter, there's no guarantee that the formatter callbacks have been called to update tref. In that case, output almost certainly will be lost.

In any case, if you are using the new function to flush the queue then the
``custom'' formatters you mentioned may never flush their underlying output
device and you will loose their output as well, no ?

Flushing the queue without flushing the device is just a source of eroor in
your ``use-case'', no ?

I don't think this is right in general. This is where an assumption about the underlying output stream (or device or concept or whathaveyou) comes into play. If the output is buffered, then your situation may arise. If the write handler of the formatter (first argument of Format.make_formatter) just writes to some buffer and there's no other thread scheduling writes, then this may happen. But then for output to be lost, the user must either omit a flush (in which case his program is wrong), or the flush handler (the second argument for Format.make_formatter) for the user's custom formatter is called and just decides to do nothing (in which his program is both wrong and bizarre).

All of this is possible before these commits were merged. In addition to that, a user that defined a custom formatter may have incorrectly believed (as I did) that the write and flush callbacks would have been called after calls to fprintf and related functions. With all output (seemingly) buffered in the output stream, all that's left is to flush the stream and wait for a response. In reality though, the output would have been lost.

In my use-case, I want to flush the internal formatter queue so I can get bytes that need to be written output stream, so I can schedule the writes. This is independent from flushing the output stream, which will block until previous writes complete. (Or at least, it's a possibility that the operation will block.) To separate those two concerns—getting bytes to schedule for writing and blocking until previous writes complete—in a custom formatter requires that formatter's underlying queue can be flushed.

This is important in order to perform efficient i/o, which I won't get into further here.

Lot of questions: if you use kfprintf, it means you want the get the
formatter in your continuation, then is'nt it to go on printing ?

The use of kprintf is so that the types works out within a library that

I'm writing, but the need is not specific to kprintf. I simply
discovered the limitation of the module through that function. The same
situation may arise through multiple calls to fprintf. In my particular
use case, there is only one call to kprintf, after which the internals
of the custom formatter are collected and returned to the user.

I do not understand what you mean by internals'' of a fprmatter, don't you simply meanthe printed material'' or are you speaking of some part of the
internal data structure of the formatter ?

What assumption am I supposed to do about the output stream attached to the
formatter ? I was not even assuming the formatter being attached to a stream!

I'm sorry but I still do not understand your use case: could you define what you
mean by a ``custom'' formatter ?

A custom formatter is the result of a call to
Format.make_formatter.

As the name indicates, the result of ``make_formatter'' is simply named a
formatter; nothing custom in the result of make_formatter...

Here's a block of code that illustrates the use
case:

let fprintf format =
  let tref     = ref noop in
  let formatter =
    Format.make_formatter
      (fun str pos len -> tref := !tref ^^ string ~pos ~len str)
      (fun ()          -> tref := !tref ^^ flush)
  in
  let k fmt =
    Format.pp_flush_formatter fmt ();
    !tref
  in
  Format.kfprintf k formatter format

Variables noop, flush, string are unbound in this piece of code.
I assumed that ( ^^ ) is the one defined in Pervasives.

However, it seems that you typically mimmick a regular formatter that prints on
a Buffer.t underlying output device. Why not using Format.formatter_of_buffer ?

May I also suggest that repeated format concatenation is not efficient ? This
is in some sense similar to string concatenation and all the point of Format
pretty-printing has been to carefully avoid that efficiency bottelneck...

The format string is used as an edsl, producing a data structure that
represents the writes and flushes that must be performed. The result can be
composed with other write and flushed operations (some of which are defined
via generator functions). Additional library code can then traverse the
representation, scheduling writes and flushes as defined by the
user. Without a call to Format.pp_flush_formatter, there's no guarantee
that the formatter callbacks have been called to update tref. In that
case, output almost certainly will be lost.

Did you mean that you are parsing the resulting format strings or even
pattern matching on their internal representation as a GADT ?

In any case, if you are using the new function to flush the queue then the
``custom'' formatters you mentioned may never flush their underlying output
device and you will loose their output as well, no ?

Flushing the queue without flushing the device is just a source of eroor in
your ``use-case'', no ?

I don't think this is right in general. This is where an assumption about
the underlying output stream (or device or concept or whathaveyou) comes
into play. If the output is buffered, then your situation may arise. If
the write handler of the formatter (first argument of
Format.make_formatter) just writes to some buffer and there's no other
thread scheduling writes, then this may happen. But then for output to be
lost, the user must either omit a flush (in which case his program is
wrong), or the flush handler (the second argument for
Format.make_formatter) for the user's custom formatter is called and
just decides to do nothing.

To be clear:
Format.pp_print_flush prints all the pending material in the formatter queue
and then flushes the underlying outpout device of the formatter by calling
the out_flush field of the formatter out functions.

The new functionality, Format.pp_print_queue would do exactly the same,
except that it does not flush the output device.

So: if the underlying device never need to be flushed the two functions
are equivalent (e.g. printing to a string or printing to a Buffer.t).

Oth: if the underlying output device of the formatter needs to be flushed,
you need to use pp_print_flush NOT Format.pp_print_queue that could forget
some material in the output device.

Hence: using pp_print_flush is always safer than using pp_print_queue.

Question: what use case needs to use pp_print_queue instead of
ppr_print_flush ?

All of this is possible before these commits were merged. In addition to
that, a user that defined a custom formatter may have incorrectly believed
(as I did) that the write and flush callbacks would have been called after
calls to fprintf and related functions. With all output (seemingly)
buffered in the output stream, all that's left is to flush the stream and
wait for a response. In reality though, the output would have been lost.

I do not understant what you mean by ``the write call back'': there is no
simlar wording in the Format documenttion.

I do not either understand how you could imagine that a call to fprintf would
flush the pretty-printer queue since fprintf calls are typically composed and
nested in pretty-printing routines. If each call to fprintf would flush the
pretty-printer queue, how composition could properly work ? How conversion %a
could ever work in this case?

(Consider for instance a polymorphic routine that prints a pair:

let print_pair pa pb ppf (x, y) =
Format.fprint ppf "@[<1>(%a,@ %a)@]" pa x pb y
;;

Printers pa and pb are likely to use Format.fprintf: if you imagine that pa
would flush the pretty-printer queue then all the boxes would be claosed
and pb would print y outside of the proper box, starting at the left margin !)

In the same vein, the need to flush the pretty-printer to get all the printed
material is clearly stated in the documentation of the module:

Rule of thumb for casual users of this library:

• use simple boxes ...
• ...
• at the end of your routine, flush the pretty-printer to display all the
  remaining material, e.g. evaluate [print_newline ()].

And also:

In case of interactive use, the system closes all opened boxes and
flushes all pending text (as with the [print_newline] function)
after each phrase. Each phrase is therefore executed in the initial
state of the pretty-printer.

Last, but not least:

Warning: the material output by the following functions is delayed
in the pretty-printer queue in order to compute the proper line
splitting. Hence, you should not mix calls to the printing functions
of the basic I/O system with calls to the functions of this module:
this could result in some strange output seemingly unrelated with
the evaluation order of printing commands.

I agree that this may still be unclear; in this case, I would be glad to
use your suggestion to enhance the documentation.

In my use-case, I want to flush the internal formatter queue so I can get
bytes that need to be written output stream, so I can schedule the
writes.

Then you definitely need to use a formatter writing to a Buffer.t output
device.

This is independent from flushing the output stream, which will
block until previous writes complete. (Or at least, it's a possibility that
the operation will block.) To separate those two concerns—getting
bytes to schedule for writing and blocking until previous writes
complete—in a custom formatter requires that formatter's underlying
queue can be flushed.

You should separate the two concerns: get the pretty-printed material bytes
on the one hand and writing those bytes out on the other hand.

Using a formater writing to a Buffer.t and using pp_print_flush asnwers the
first question. Once you get the bytes in the Buffer.t value, you can write the
bytes when you want, where you want and the way you want.

This is important in order to perform efficient i/o, which I won't get into
further here.

I will not either.

Regards,

Pierre Weis

INRIA Paris, http://bat8.inria.fr/~weis/

My understanding of @seliopou's setting is the following:

• The user-defined formatter is not directly writing to an I/O device, but building an abstract representation of the I/O calls to be performed later. This abstraction representation contains both writes and flushes requested by the user.
• The distinction between pp_print_flush and pp_print_queue in this use-case is that pp_print_flush is what is used by the user to explicitly demand a flush of the underlying I/O device (this is why the second argument of make_formatter must explicitly insert a flush in the abstract representation), while pp_print_queue is what @seliopou would need to make sure that the abstract representation is complete before evaluating it. Using pp_print_flush in this case would insert extra flushes on the underlying I/O device, and there are performance worries about that.

I can't comment on the performance aspects of the discussion, but to me this seems to be a reasonable use-case for making a distinction between those two notions of flushing. If I understand correctly, what is needed here is a "soft flush" that closes all boxes and perform the remaining formatting actions, without explicitly calling a "hard" flush on the underlying device.

but to me this seems to be a reasonable use-case for making a distinction between those two notions of flushing. If I understand correctly, what is needed here is a "soft flush" that closes all boxes and perform the remaining formatting actions, without explicitly calling a "hard" flush on the underlying device.

This is a very good summary of the issue and why @seliopou implemented this in the first place (after a discussion with me).

Also, I would like to point out that pp_flush_formatter is not unsafe, as long as you actually document what it does (closing all boxes). The issues with bprintf that makes it confusing is that the box closing behavior is not documented.

Unless there is another way of implementing soft flushing, I think this function should be kept as it's currently implemented.

I'm still not convinced by the new pp_print_queue functionality, but the
use-case exhibit a real concern.

I propose to add a new facility to support the use case directly within
module Format.

This message is long: the short version is:

A: Adding a facility to produce some abstract data instead of printing,
could be a useful addition to Format.

B: Adding pp_print_queue is easy, but we refrained to do so for years, since
this is also a bad idea.

C: Use-case: You do not need pp_print_queue to produce the abstract syntax:
pp_print_flush would do it with no extra cost.

D: Use-case: fprintf flushes the pretty-printer queue at each call: this
is a bad idea.

This message is long: the long version is:

A: design a formatter to pretty-print in an abstract way

Abstract printing, i.e. producing some data structure instead of printing to
some I/O device, or string, or buffer has already been proposed for the
module Format, This was supposed to produce the box structure together with
string tokens and break hints (and the semantics tags and so-on).

This was ambitious and complex and was never implemented, We could now
implement a simpler version that would simply record an abstract version of
the I/O commands required to display the pretty-printed material.

To provide this facility, we just have to design a type specification of the
abstract version of I/O commands and implement a formatter that would produce
such data (let's call it an abstract formatter).

Using a formatter will be conservative to keep the compositionality and
incrementality good properties of pretty-printing with Format. Last but not
least, no need to design and use a special purpose printf-like function: the
user would simply call regular pretty-printing functions with the new
abstract formatter.

B: pp_print_queue is a bad idea

I already mentioned that pp_print_queue would have a bizarre functionality:
it terminates the pretty-printing session but complete output display is not
granted. Moreover, specifying in which case complete output display will
occur is complex and program dependent (at worse, it depends on the output
material).

C: use-case: use pp_print_flush to produce the abstract representation

If your special-purpose pretty-printer produces an abstract representation of
the I/Os to be performed, then pp_print_flush would not perform any low level
flush: it would flush the pretty-printer queue as usual but then produce the
corresponding abstract representation of doing a flush. Only the second phase
would perform true bytes output and only the second phase would flush when
and where it should.

To be clear, using pp_print_flush on the formatter will ensure that the
abstract data structure is complete: no low level flush can occur since you
are just producing an abstract data structure.

D: use-case: flushing the pretty-printer at each call is a bad idea

@seliopou: your special purpose fprintf function flushes the pretty printer
at each call. This is not a benign property: it drastically modifies
fprintf's scope of application and lowers its user-friendliness!

For instance, the special-purpose fprintf function cannot support multiple
related calls, since each successive call would flush the queue.

Also, this behavior forbids composition of pretty-printing routines using it.
Consider for instance pretty-printing a pair of values, using:
let print_pair pa pb ppf (x, y) =
fprint ppf "@[<1>(%a,@ %a)@]" pa x pb y
if pa and pb use the special purpose fprintf function, the output would be
wrong!

This ``each call flushes'' behavior was precisely the motivation to deprecate
the bprintf function: flushing the queue at each call renders bprintf almost
unusable in practice. In contrast, using a formatter writing to a Buffer was
clearer and restored the magic.

@pierreweis if you'll notice, my fprintf function only flushes the queue once since the formatter that is flushed is created within the body of the function, and does not escape that function body execution. This is by design. It's all I need.

If the need is clear, and the behavior of the function can be properly documented, what's the problem? Users don't read documentation and misuse APIs all the time. That's not a reason to not write the API to begin with. And practically speaking, if this function is included, I imagine it will go mostly ignored by casual users.

I carefully read your code and I understand it. I'm sorry to insist
introducing a flush queue function is a bad idea: the resulting
pretty-printing routines are not compositional, since each call to your
printf function flushes the queue.

I refuse to introduce a function that is useless and error prone.

I know that users may misuse APIs and that's exactly the reason why I do not
want to introduce a feature that encourages bad usage and leads to errors.

As I said, I prefer to introduce a way to print abstractly: this feature is
compositional and fits nicely in the library. It will provide the
functionality you want: printing without I/Os.

@pierreweis if you'll notice, my fprintf function only flushes the queue once since the formatter that is flushed is created within the body of the function, and does not escape that function body execution. This is by design. It's all I need.

If the need is clear, and the behavior of the function can be properly documented, what's the problem? Users don't read documentation and misuse APIs all the time. That's not a reason to not write the API to begin with. And practically speaking, if this function is included, I imagine it will go mostly ignored by casual users.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#506 (comment)

@seliopou @pierreweis There seems to be stalemate on this pull request. What is going to happen to it?

@mshinwell The goal was to provide the needed tools in PR #615. There have been some interesting comments there but not from @seliopou.

#615 has been merged, so I think this can be closed.