[core][observability] Logs are duplicated if multiple nodes are running on same machine

[kevin85421]

Why are these changes needed?

Each Ray node has a LogMonitor, which watches logs under /tmp/ray/$session_name/logs and publishes them to GCS so that the driver process can access the logs.

However, when we launch two Ray nodes on a single host, both LogMonitor watch the same directory /tmp/ray/$session_name/logs.

Related issue number

Closes #48642

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

# example.py
import ray

@ray.remote
def foo():
    print('hello')


if __name__ == '__main__':
    ray.init()
    handle = foo.remote()
    ray.get(handle)

• Without this PR
  

• With this PR
  

[jjyao]

This file won't exist if redict_logging is False?

[kevin85421]

In which case will this happen?

Case 1: Do you mean setting RAY_LOG_TO_STDERR=1 for the head node and RAY_LOG_TO_STDERR=0 for the worker nodes? I am not sure what the expected behavior is for this. Should we redirect the logs to the driver process or not?

RAY_LOG_TO_STDERR=1 ray start --head --include-dashboard=True
ray start --address=127.0.0.1:6379

# Try to schedule the task to the worker node
python3 example.py

Case 2:

ray start --head --include-dashboard=True
ray start --address=127.0.0.1:6379
RAY_LOG_TO_STDERR=1 python3 example.py

log_monitor.log will be created by ray start --head --include-dashboard=True.

[kevin85421]

RAY_LOG_TO_STDERR=1 ray start --head --include-dashboard=True

The above command will create the following log files. However, checking other log files seems a bit weird.

[kevin85421]

If we plan to handle this case, how about creating an empty file named is_head in the log directory? The worker node can check whether the file exists to decide whether to launch the log monitor process.

[kevin85421]

Updated