I'm not sure in how many packages lwt should be split -- the log sublibrary is used by both preemptive and unix -- thus only once these pieces are moved into separate packages, there is an actual effect. before that it is dangerous (if split away it introduces circular dependencies). OTOH unix and preemptive can be rewritten using the Logs library (or to not care about initialising loggers, but use Logs, and rely on applicaions to setup reporters as appropriate), but this will likely break the API. Any opinions? I always failed to use Lwt_log, and am pretty happy with the Logs library myself.

There seem to be two kinds of interactions between Lwt_log and the rest of lwt.unix/lwt.preemptive:

1. lwt.unix uses Lwt_log for actual logging. I think all of these should be removed. Lwt shouldn't do any logging.
2. Lwt_daemon actually has a non-trivial interaction in the API. I don't know immediately what to do about this. I guess it would prevent factoring out lwt.log.

For (1), I see only two trivial logging expressions (in lwt_throttle.ml and lwt_timeout.ml). AFAICT, lwt.preemptive doesn't actually ever log anything, even though the ability to set a logging function is provided in its API, and it references Lwt_log in its default logging function (which, again, AFAICT is never called).

In summary, I'd be happy to get the process that this PR would start, actually started, assuming we can figure out what to do about Lwt_daemon :) Otherwise, I think we will have to delay it for now, by default.

Lwt_daemon may be worth spinning off into its own package as well. It works for basic uses but could use some extra attention from what I recall.

Yeah, factoring out Lwt_daemon might be the only way to shed Lwt_log.

I really don't like the idea of creating new packages just to deprecate them, or factoring out a package with one module that has one function, but I can't think of anything better.. and it will do the trick.

Is there maybe some naming convention or opam field we could use or create, for marking such packages? Like lwt_camlp4_deprecated, or deprecated: true? Edit: cc @avsm, @AltGr.

What's slowing me down right now is that I think Lwt_daemon is really helpful for Lwt. Like, it is not obvious how to write an Lwt-friendly daemonize from scratch. So, we will need to address this either with excellent docs, or with a new daemonize (in Lwt_unix?) that doesn't touch Lwt_log. If we go the latter route, maybe we should include (EDIT: the current) Lwt_daemon in the lwt_log package, even though it will seem kind of weird.

I looked through opam packages depending on Lwt, and found:

• arakoon, charrua-*, message-switch use Lwt_daemon, but in a way that does not interact with Lwt_log at all.
• usbmux, xentropyd depend on the ?syslog parameter of Lwt_daemon.daemonize.
• yurt depends on the ability to redirect to loggers.

So cc @toolslive, @domsj, @mseri, @gaborigloi, @fxfactorial, @djs55, @zshipko for any opinions on gently moving Lwt_daemon to a new package lwt_log (or lwt_log_unix, see below).

A large number of packages depend on Lwt_log itself, but I'm pretty confident we should factor that out. It's going to be a bit of a pain, as there are non-Unix and Unix layers, so perhaps we will need two packages. We should consider that now, so we can create the right package names. @Drup, any opinions on this based on Ocsigen experience with Lwt_log and Unix/"pure" packages?

The plan would be:

• Create a "proxy" lwt_log package with Lwt_log, Lwt_log_core, Lwt_daemon so users can update their deps starting with Lwt 3.2.0 (done by this PR).
• Remove all dependencies of Lwt on Lwt_log besides by Lwt_daemon.
• Make lwt_log a "real" package in 4.0.0, and move the above modules to it. For users that update their deps, the transition should be smooth. We will notify during release of 3.2.0 the maintainers of all opam packages that depend on any of the code to be moved.
• On demand, if someone wants to have a daemonize function that does not come from lwt_log, we will add Lwt_unix.daemonize that is not aware of loggers.

There is one more complication, which is that the PPX and the Camlp4 syntax have some support for Lwt_log. However, I think it is indirect. A user of those features can depend on the new lwt_log/lwt_log_unix to make sure the generated code compiles and links.

What's the purpose of spliting lwt_log out ? I doesn't really bring extra dependencies or anything, and it's not like it's difficult to build.

@Drup, to deprecate it and encourage use of logs or other libraries. See #468 (comment) and further. Also to make the mapping between opam and Findlib packages more direct.

The "broader" picture is I'd be happy to see work on Lwt_log, but I don't think there is enough labor available for working on Lwt in general, and there are much higher priority Lwt issues and projects besides Lwt_log. As a result, a higher quality can be more easily achieved in libraries outside Lwt. So I prefer to focus on the core of what Lwt does (promises and I/O for now), and shed anything else if there are good alternatives.

@aantron Alba is a dependency too. Anyway, the above plan looks fine, and we'll adapt when we upgrade our dependency chain.

Another point to move daemonization to its own package is that it really shouldn't be encouraged either. It doesn't really serve a purpose when you have a modern init system http://blog.benoitblanchon.fr/linux-daemon-considered-harmful/

It seems to also be deprecated on macOS, but I'm not sure about "obscure" systems.

The "broader" picture is I'd be happy to see work on Lwt_log, but I don't think there is enough labor available for working on Lwt in general, and there are much higher priority Lwt issues and projects besides Lwt_log. As a result, a higher quality can be more easily achieved in libraries outside Lwt. So I prefer to focus on the core of what Lwt does (promises and I/O for now), and shed anything else if there are good alternatives.

This sounds like an excellent approach to me. When Lwt_logs was first created, there were very few alternatives, but that is no longer the case. There are also quite a few libraries available for OS-specific launching now (e.g. ocaml-launchd for osx) as an alternative to daemonising, as @rgrinberg notes.

Your plan makes perfect sense and I welcome the split and the cleanup.

We use the daemon and the logs in few places but somewhat inconsistenly. We have one more package that is using Lwt_daemonize and Lwt_log: https://github.com/xapi-project/wsproxy/blob/master/cli/wsproxy.ml#L116 but I don't think that would be massively affected either

@avsm @rgrinberg What libraries would you recommend as daemonising alternatives on linux? We are using ocaml-systemd in some places, but we couldn't use it everywhere due to undebuggable segfaults in some cases

Ok, I rebased this PR (there was a merge conflict). I added more commits, such that:

• @Drup is no longer listed as the author of Lwt_log, per his objection.
• There are two proxy packages created, instead of one. They are lwt_log.core for modules Lwt_log_core, Lwt_log_rules (the pure-OCaml code), and lwt_log.unix for modules Lwt_log, Lwt_daemon (the Unix stuff). This reflects a pre-existing split in the log library in Lwt.
• lwt_log is an alias for lwt_log.unix. This could be surprising, because lwt.log corresponds to lwt_log.core, and users that wanted what will be lwt_log.unix currently have to depend on lwt.log and lwt.unix (extra credit if you can actually follow that sentence and distinguish underscores from periods very easily :p). However, I believe the current setup is the more surprising one, I would expect depending on lwt.log to give me module Lwt_log, but it doesn't.
• I added a usage test to make sure things remain working in the future.

If there are no objections, I will squash-merge this (so @hannesm will be the author, I will be the committer, and there will be a link from the commit message to this PR for the details).

That will be the first step of factoring out lwt_log, and will be released in 3.2.0.

In a previous commit, 4a7cf8c, now in master, I removed all other dependencies on Lwt_log in lwt.unix, besides the one in Lwt_daemon.

@aantron thanks for taking care of that! highly appreciated!

Actually, I think it's better to eliminate lwt_log.unix and have only lwt_log (not as an alias), to keep things simpler for Jbuilder.