Conversation
openpype/modules/deadline/repository/custom/plugins/GlobalJobPreLoad.py
Outdated
Show resolved
Hide resolved
openpype/modules/deadline/repository/custom/plugins/GlobalJobPreLoad.py
Outdated
Show resolved
Hide resolved
BigRoy
left a comment
There was a problem hiding this comment.
Aside from code cosmetics the feature makes sense.
I wonder whether we should add a comment in the code in the if branch WHY we're doing it. E.g.
# Set os.environ[PATH] so studio settings' path entries
# can be used to define search path for executables.
|
Actually, one other note, can we add documentation about this use case? Somewhere in Admin Docs for Deadline we should explain how to configure things to allow Application Versions to be completely picked by settings using this method. |
openpype/modules/deadline/repository/custom/plugins/GlobalJobPreLoad.py
Outdated
Show resolved
Hide resolved
|
I followed your testing notes to submit the rendering. It fails rendering and gets the error on deadline. |
Yes, if your Deadline doesn't have the parameters required for a Houdini version to begin with it'll fail. Not sure how much Deadline still releases updates since it's gone "free" but I reckon you're best off manually adding the parameters if it's not available in an update yet. |
|
I have added the 19.5 param into Houdini. And it still fails rendering but it successfully renders when it is with the full path.
The log below is the error shown. |
|
@moonyuet There should be there a line that prints |
I find out that I have disabled the ProcessSubmitPublishJobOnFarm for testing, will be testing the PR with that being enabled. |
|
It still errors out even if I turn on the submit publish job on farm. |
|
Could you share your log for the FIRST task your machine does on that machine? The GlobalJobPreLoad only runs before the Job starts and not before every task - and thus the log of the GlobalJobPreLoad itself would only be visible on logs for a machine that is starting the job and not whenever it goes to the next frame on a job as far as I know. That's also why your shared logs I believe are lacking any logs/prints from the GlobalJobPreLoad script. In short, make sure to check the logs for the FIRST TASK a machine starts the job with. Also, make sure to 'resubmit' the job whenever you update the |
This is the error log for the first task(task id 0) I have tried the re-submission too... |
|
Could you please post the full log? :) You seem to be cutting of the part of the log that'd actually be showing the output of the GlobalJobPreLoad logs - which tend to be further down. That the error is the same I understand, but I'm mostly interested in seeing/reading what GlobalJobPreLoad ends up doing. What Also, minor side note that task id being zero doesn't necessarily mean that your machine STARTED the job on that task. It could've been running the job on tasks 0, 1, 2 and then you requeued 0 as it was running and now it would process task id 0 again but it'd still be a continuous job run from 1, 2 if it was still rendering the job. ;) |
It doesn't print PATH in the log. |
|
No the You seem to be lacking ANY output from the You should at the very least see the print output of these two lines and thus should see in your logs: That is no matter what plugin you are running and no matter how your job is submitted through Deadline - if that It's good to note that HOW you've configured here is actually wrong too. You don't want to set that absolute Houdini path in the PATH in the global env for Houdini application but set the correct Houdini path per version. In your case since you have {
"PATH": [
"C:/Program Files/Side Effects Software/Houdini {HOUDINI_VERSION}/bin",
"{PATH}"
]
}Note: You do not replace |
|
Repeating here what I said on Discord, not all infrastructures will have the same environment for Deadline workers than the local one thus why we would want to keep full control of the PATH used in Deadline versus what's used locally. Our Deadline workers for example are running as Dockerized containers and the binaries location where the DCCs are installed doesn't match the PATH of the workstation that submitted the job. I don't think you should be merging this change and someone could argue that the GlobalJobPreLoad is actually already injecting too many env vars into the worker and we should control what gets passed (i.e., PYTHONPATH could cause problems as well if taken from the local workstation) |
It could just be a toggle in the Deadline Plugin Settings by the way - trivial as that.
Again, we're not passing the source machine's environment. We're injecting the Application environment that OpenPype's Application launching builds. That environment is computed locally and build locally on the machine, not passed along from where it got submitted at all. Is that maybe a misinterpretation you've made? I feel like I understand your worry if we're just randomly injecting user environments to the farm but in practice this PR does nothing than just also adding
For completeness sake, here's that discord post for future reference. |
Ohh you are right, I had a lapsus thinking where these env vars were being passed, I thought these were being set as the job properties of Deadline and could affect other processes. Yeah that makes sense, sorry for the confusion, should be okay to merge |
It seems that I am still using the old version of repo for that....(I forgot to copy the new GlobalPreJobLoad.py into the custom folder to replace the old one) |
…lJobPreLoad-to-update-job-env
Changelog Description
This PR makes
GlobalJobPreLoadto setPATHenvironment in deadline jobs so that we don't have to use the full executable path for deadline to launch the dcc app.This trick should save us adding logic to pass houdini patch version and modifying Houdini deadline plugin.
This trick should work with other DCCs
Example :
To make deadline call the desired Houdini executable, we should :
update
PATHenvironment in the app environment and addHOUDINI_VERSIONenvironmentin deadline plugin settings only use executable name
Additional Info
The idea of this PR was discussed here #5384
Testing notes:
Here are snippets for maya and houdini, but feel free to do the same with other DCCs
for houdini, you can use
for maya, you can use
for houdini, you can use
for maya, you can use