Houdini: Submit houdini patch version to deadline#5384
Houdini: Submit houdini patch version to deadline#5384MustafaJafar wants to merge 11 commits intodevelopfrom
Conversation
|
I'm not a fan of including our own custom full Houdini plug-in for Deadline. It can easily get outdated compared to Deadline updates and I suspect it'll be hard to keep track of what changes were made for OpenPype and which were made due to being outdated - especially if currently all that's changed is the I'd say instead of this, couldn't we instead just rely on |
I'm not sure how to tell deadline to resolve
I think this will require modifing HythonExecutable def RenderExecutable( self ):
version = self.GetPluginInfoEntryWithDefault( "Version", "18.5" ).replace( ".", "_" )
return self.GetRenderExecutable( "Houdini" + version + "_Hython_Executable", "Houdini " + version)SimTracker version = self.GetPluginInfoEntryWithDefault( "Version", "18.5" ).replace( ".", "_" )
# Retrieve houdini simtracker executable
simTracker = self.GetRenderExecutable( "Houdini" + version + "_SimTracker", "Houdini " + version + " SimTracker")Maybe we can just send the executable path and tell def RenderExecutable( self ):
hou_exe = self.GetPluginInfoEntryWithDefault( "Houdini_exe", "" )
if not hou_exe:
version = self.GetPluginInfoEntryWithDefault( "Version", "18.5" ).replace( ".", "_" )
hou_exe = self.GetRenderExecutable( "Houdini" + version + "_Hython_Executable", "Houdini " + version)
return hou_exe same for |
|
Ah, yes. That doesn't help too much then either. It's too bad we need this much 'changing' of Deadline native functionality. It basically means we'll now also start maintaining Deadline's Houdini plug-in with OpenPype which is not a nice thing to have added on our to-do lists. 🗡️ |
|
Man I don't like any of this to be honest, but it is actual scenario. I haven't tried myself, but @mustafa-zarkash can we make it so it's backwards compatible with default houdini deadline plugin? The point is that for majority of studios major.minor is probably enough, considering we've only had one request for this and I'm no too keen on maintaining the full plulgin all the time. We could possibly just make sure we're able to send the patch version and give studio the option to maintain their own deadline plugin. It might accidentally creep the scope of houdini deadline maintenance for us. |
|
@mkolar I agree. I think adding a toggle some where in settings so that admins can tell deadline publisher to send patch version or not with a big note that enabling this requirs modifying houdini.param with a mini guide about how to do so. |
|
So I just learned that Deadline has a "Rez" plugin apparently, here are Deadline + Rez plugin. It uses the On Job Submitted event just to insert some Rez things to 'configure' things alright for the to be launched application. I suspect since Rez basically also builds up an environment that could supply e.g. a Houdini version that it might've tried to solve a similar issue? Maybe it's a decent reference at least as to what we can do without touching the Houdini plugin itself. I would expect to be able to do something similar where we can e.g. in GlobalJobPreLoad or whatever initialize the relevant Application environment (or make the executable accessible in a way for example add the executable's folder to start of That way we can influence how Deadline launches houdini, but don't need to modify Houdini plugin itself. |
|
@mkolar @BigRoy I don't know what optional and active will do but I think they will be used by
|
|
i tested with the 19.5.435 version of Houdini It doesn't work and gives the error below. (I guess this one is still in draft, right?) |
To check if this PR is working we need to check the job parameters.
The thing is: by default deadline will complain about it can't find the houdin exe In short: these blocks should added to Hython Executable blockSim Tracker blockAfter Updating that file, you will find new entries to add houdini exe |
Would be great if you can have a doc to tell the 19.5 users how to add the params into the Houdini (which file to edited etc.) so that it would update the plugin configuration. |
|
@moonyuet I have updated the docs. |
I just saw your comment now. Adding param can be messy if the users modify Houdini plugin especially when switching from different projects with different Houdini versions(some renderers such as Arnold and Redshift can only be used in certain patch versions). I need to download two different versions of Houdini to test things and I have to make sure to download the right minor version. |
|
I'm wondering now
|
|
More details here.
Hmm, not really. Sort of. Usually a process started from the plugin might inherit the environment from the Deadline plugin too - but it's not ALWAYS the case because a plugin can disable that behavior for processes it executes, see That's why I recommended the approach to keep the original behavior (still use if "PATH" in contents:
deadlinePlugin.SetEnvironmentVariable("PATH", contents["PATH"])Which I believe is quite similar to: if "PATH" in contents:
os.environ["PATH"] = contents["PATH"]Except that Actually with that said, if
Yes, I'm quite certain it will.
Yes, GlobalJobPreLoad runs before the render plugin for the job starts.
It does return the full path to |
|
@BigRoy , I did a check for the env by adding the test script(the script I used for checking the remote_publish but it still checks the sys.path) into The script I used. Maybe the env can be set up by adding the script which sets up the environment variable for {PATH} as PreJobScript in Houdini deadline submission. It will do the job right before all the rendering actions. |
Note that Anyway, we're not looking to adjust Other than that I have no idea what you're trying to do with the running it as pre job script. |
i dont mean they are the same. maybe i should explain better, what i mean is like it seems that both sys path and os.getenv("path") (see mustafa's comment) dont store the hython directory coming from the related environment set in the OP settings.
I can check with logging but i dont really think it is there. For prejob script, I mean we can use the prejobscript to set up the environment for setting up the path for hython exeuctable |
As long as this proposed solution will work with other DCCs, then we should discuss it a in different place like This feature request is Houdini specific, and I really don't know if it's needed to have a general solution that works with other DCCs
I agree with Kayla we need to find a solution that works in the first place to satisfy the current need
I think having the two options will be nice, which adds some flexibility as admins would be able to choose from two solutions
|
…version-to-Deadline
…dline' of https://github.com/ynput/OpenPype into enhancement/OP-6381_submit-houdini-patch-version-to-Deadline
Have you tried changing the OpenPype If that works for you it's really the simplest use case.
I've seen many beginners trip over the Deadline Param files (and even breaking them hehe). We'd need to clearly document how to do it, but I'd say if we need to do that anyway the other more generalized way I described is just as clear and doesn't require us to maintain a custom Houdini params/plugin, etc. |
I'm going to try this. |
|
Hey, @BigRoy Setting PATH in OP settingsHere are my Openpype settings. I printed Setting Deadline param file
Modifying
|
|
Your second printed log shows the issue. The environment you have there in I don't see the Houdini folder there?
No. These must be computed at runtime on the farm machine to be cross os compatible, etc.
GlobalJobPreLoad runs before the render plugin so should be on time. |
yes it does not include the app environment |
That'd mean the Probably @iLLiCiTiT knows why it might exclude the Application Settings' "environment"? |
I followed that thread and found that it eventually calls so I gave it a test in houdini import os
from openpype.lib.applications import (
get_app_environments_for_context,
LaunchTypes,
)
project = os.getenv("AVALON_PROJECT")
asset = os.getenv("AVALON_ASSET")
task = os.getenv("AVALON_TASK")
app = os.getenv("AVALON_APP_NAME")
env_group = "farm" # "standard"
env = get_app_environments_for_context(
project,
asset,
task,
app,
env_group=env_group,
launch_type=LaunchTypes.farm_render
)
print(env)and I could find in the results Also, I print |
|
I believe the function also returns your local environment (it includes it) and thus if you already run it within Houdini it'd include any env vars already set there - that's also why you're seeing the You should really be running this from say the Console in the OpenPype launcher (but since that doesn't have the asset env vars, etc. pass it actual names you know exist) |
I can find it in OP Console too!
|
|
I suspect this will be a question for @iLLiCiTiT |
@BigRoy and after fixing that, Then, I added these lines just after these other line which prints fine! but deadline still fails |
|
@MustafaJafar This is perfect - so close. I've decided to do a test run on my end. This I how I set the Houdini plugin settings in Deadline: Then I skipped OpenPype completely and just did a 'deadline test' and submitted a Houdini scene through the Deadline Monitor for that Houdini version. Next this is what I tried in import os
HOUDINI_LOCATION = r"C:\Program Files\Side Effects Software\Houdini 19.5.435\bin"
path = os.pathsep.join([HOUDINI_LOCATION, os.environ["PATH"]])
print(f"Testing deadlinePlugin.SetEnvironmentVariable: {path}")
deadlinePlugin.SetEnvironmentVariable("PATH", path)Which failed. Then I tried: print(f"Testing deadlinePlugin.SetProcessEnvironmentVariable: {path}")
deadlinePlugin.SetProcessEnvironmentVariable("PATH", path)Which failed. Then I tried: print(f"Setting os.environ['PATH']: {path}")
os.environ["PATH"] = pathWhich worked. ✅ So the solution is to just add it to The logs showed: Could you try that @MustafaJafar |
|
It's good to note that Deadline has quite a few settings involving "environments" of the jobs. I'm not sure how any of these influence the behavior: Sandbox Plugins - it might be that this alters the behavior when disabled? Render as user - I'm not sure if that user would then similarly inherit the environment. I suspect it should and that it runs even the deadline plugin and globaljobpreload as that user from the get go so then works fine. It's quite an advanced feature though - we don't use it. |
|
@BigRoy
We can test that in the other PR
|
|
This PR is closed in favor of #5622 |
Changelog Description
By default houdini deadline submission includes
major.minorversion numbers onlyAs requested, to include patch version as follows
major.minor.patchAdditional Info
This modifications require to update houdini deadline plugin param file.
I have added a mini guide in docs
Testing notes:
DeadlineRepo\plugins\Houdini\Houdini.paramas suggested