Why not just using glog for writing framework? What's the reason why we'd like to develop our own file writing framework?

Why not just using glog for writing framework? What's the reason why we'd like to develop our own file writing framework?

First, I aslo disagree with repeated development of file systems in ray. But glog has low customizability, so we have to face these:

• No rotating (we need automatically clean log files genreated by long running jobs),
• Can be initialized only once,
• No custom format pattern.

I'd like to replace glog with spdlog https://github.com/gabime/spdlog/tree/v1.x/include/spdlog (popular and good performance).
Or making spdlog as an option.

@Basasuya LGTM. Could you check the rest of comments and reslove them.

@rkooo567 @raulchen @yuyiming Could you merge it if travis is green.

@Basasuya More comments should be better for more readable and advance development before all public functions.

I will review this very soon. Have been too busy by Ray 1.0 related stuff.

I have added some code comments in this PR

Hey guys, do we really need spdlog here? I would much rather keep glog, even if it has slightly less features, rather than add a new dependency on some random library.

Hi eric, my answer is yes. We do need spdlog becasue of its features and performance.

In our production business and best practices, we not only need to support a large number of long-running jobs, some are even online jobs. These long-running tasks are usually run in a docker or serverless environment, and their specifications are destined to not carry large-capacity disks. Therefore, we must clean up the log files regularly to prevent our long-running tasks from being affected.
Besides, it is not that we have to replace glog, because glog is currently unable to meet business needs.
Let me briefly summarize again:

1. Cannot automatically clean up and rotating the log.
2. There are two duplicate outputs, one is standard output and the other is log file.
3. For each process, glog can only be initialized once. If not, multiple errors are reported and the process exits abnormally. This means that if the user uses cpp worker, the log generated by the user's actor logic will be printed in the same log file as the core worker. Event sender can not create a new file for genrating event stuff as well.
4. The log level cannot be dynamically modified at runtime, which is very helpful for debugging long-running tasks.
5. Unable to customize the log input format, which is fatal for events.

We have also discussed for a long time whether to introduce this dependency. The conclusion is that we firstly introduce spdlog into the event system so that its needs can be met, instead of modifying glog or repeating filesystem operations.
The next step is to add a switch to select the logging tool. So we can freely choose whether to use glog or spdlog. The design of the RAY_LOG macro allows us to realize these two kinds of compatibility easily.

In addition, it is worth mentioning that the maintenance of glog is indeed inferior to spdlog. My understanding is that adapting to the needs to make some changes will make ray a greater open source software. Instead of avoiding it, endure it temporarily.

cc @Basasuya @raulchen @jovany-wang

@ericl @rkooo567 @ashione We have discussed this in slack
https://ray-distributed.slack.com/archives/C016MPZE1L1/p1598937145003600

https://stackoverflow.com/questions/16439857/google-glog-rolling-files seems to indicate glog supports log rotation, have you looked into this?

@ericl I think the hardest blocker for using glog is 3 (imo, other features are not "absolutely needed" for OSS). For events, we'd like to write them in a separate file from regular logs.

It seems like there's a way to implement additional log sink for glog, but it looked a little sketchy (https://ray-distributed.slack.com/archives/C016MPZE1L1/p1598943141006800?thread_ts=1598937145.003600&cid=C016MPZE1L1).

Hmm I can understand the desire to write to a separate file, but why not instead just write at DEBUG or INFO level into the normal logs?. It is useful to correlate these events with other log message, and easy enough to filter with grep if needed. Is the concern the volume of logs is way too high?

Hmm I can understand the desire to write to a separate file, but why not instead just write at DEBUG or INFO level into the normal logs?. It is useful to correlate these events with other log message, and easy enough to filter with grep if needed. Is the concern the volume of logs is way too high?

It is better not to mix the information of event with the normal log, otherwise this is not the so-called event, but a special log item. The purpose of putting the event information into a separate file is to facilitate the distributed retrieval or index building of dashboards or other processes. And you said that using grep to find is our way of positioning artificially.

https://stackoverflow.com/questions/16439857/google-glog-rolling-files seems to indicate glog supports log rotation, have you looked into this?

The current version of GLOG does not support rotation because it produces a random file with the current timestamp as the suffix. On GLOG's GitHub homepage, I observed that there was a PR about rotation a few years ago, and it has not been merged. Internally, we modified GLOG to achieve it.

BTW, the parameter of GLOG_max_log_size is only for generating a new file, but old files can not be deleted.

@ericl I think the hardest blocker for using glog is 3 (imo, other features are not "absolutely needed" for OSS). For events, we'd like to write them in a separate file from regular logs.

It seems like there's a way to implement additional log sink for glog, but it looked a little sketchy (https://ray-distributed.slack.com/archives/C016MPZE1L1/p1598943141006800?thread_ts=1598937145.003600&cid=C016MPZE1L1).

This way of adding custom sink also requires us to manage files and decide when and whether rotation is needed.

@ericl I don't think we should use glog at DEBUG or INFO level to send event messages. The first reason is that glog does not support multiple logger object of one thread. In this case, RAY_EVENT and RAY_LOG will share glog object, then glog can not write to different files. Second, If we redirected DEBUG, FATAL mode to write events files, these two severities are not enough in our practical experience. At the same time, it will affect the function of RAY_LOG.
In summary, glog doesn't support multiple logger object of one thread natively. And glog can not rotate file just like other log frameworks such as log4j, spdlog. That's the reason why we want to try another framework.

@rkooo567 Could you please review my reply to your previous comments?
#10419 (comment)
#10419 (comment)
#10419 (comment)