[WB-8588] Add username to debug-cli log file to prevent conflicts of multiple users#3301
[WB-8588] Add username to debug-cli log file to prevent conflicts of multiple users#3301
Conversation
ea3395f to
207e3c2
Compare
There was a problem hiding this comment.
i think this is going to add too much noise in the wandb_dir. (it is bad enough with runs.. but every cli log will be too much noise i think)
maybe a directory where the randomized log goes. and then if supported a symlink from the run dir named the original debug-cli.log... This will clobber the previous symlink but all the logs will be in the wandb/cli-logs/ directory.
The current structure:
rwxr-xr-x 41 jeff staff 1312 Feb 25 12:02 ../
-rw-r--r-- 1 jeff staff 0 Feb 23 13:15 debug-cli.log
lrwxr-xr-x 1 jeff staff 52 Feb 23 14:25 debug-internal.log@ -> run-20220223_142515-2k3yz9hl/logs/debug-internal.log
lrwxr-xr-x 1 jeff staff 43 Feb 23 14:25 debug.log@ -> run-20220223_142515-2k3yz9hl/logs/debug.log
lrwxr-xr-x 1 jeff staff 28 Feb 23 14:25 latest-run@ -> run-20220223_142515-2k3yz9hl
drwxr-xr-x 5 jeff staff 160 Feb 23 13:13 run-20220223_131333-2m5racrz/
drwxr-xr-x 5 jeff staff 160 Feb 23 13:13 run-20220223_131333-2p8nnqyq/
drwxr-xr-x 5 jeff staff 160 Feb 23 13:13 run-20220223_131333-2v4yzlgz/
drwxr-xr-x 5 jeff staff 160 Feb 23 13:13 run-20220223_131333-3k5kmcti/
We actually do a similar thing with sweeps when running locally. (iirc)
sweep-....
latest-sweep@ -> sweep-something/
Codecov Report
@@ Coverage Diff @@
## master #3301 +/- ##
==========================================
+ Coverage 81.62% 81.67% +0.05%
==========================================
Files 236 236
Lines 29108 29113 +5
==========================================
+ Hits 23758 23779 +21
+ Misses 5350 5334 -16
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There was a problem hiding this comment.
How about instead creating a user-specific log? E.g., get the current user, and append that?
import getpass
user = getpass.getuser()Otherwise, I agree with @raubitsj, that will add too much noise.
|
@raubitsj the issue with only having one file is that it prevents multiple users from running on the same machine at the same time. if they both try to wandb.login for instance, it will fail out. Looking deeper into it, it looks like there may still be additional pieces that would prevent two users from being able to run at the same time anyway. @dmitryduev I wonder if all logs should be scoped and permissioned to the user that is running? Also another question for both of you, is there anything else that may prevent two users from running at the same time? |
I was suggesting multiple files. But a symlink to the most recent one. We do this for our other logs. It should be safe |
|
@raubitsj Makes sense about symlinking to latest file. I think the symlink would still need to be scoped to the user as @dmitryduev mentions above. The main problem I'm trying to solve is multiple users running on the same machine, which I now realize might be a bigger conversation than just log files. Happy to open up a separate issue if it makes more sense to have that conversation there. |
|
I can't run any cli command on our cluster due to this.. |
|
@manuel-delverme you can set the mkdir /tmp/$(whoami)
export TMPDIR=/tmp/$(whoami) |
|
CC @dmitryduev @raubitsj let's make sure a fix for this makes it into the next release. I would think scoping the file to a well known name that's unique per user on the machine should do it. |
|
Thanks @dmitryduev looks good to me. I can't approve my own PR, but approve the changes. |
Fixes WB-8588
Description
Testing
How was this PR tested?
Checklist