DYN-9388: Draft - New PythonNet3 engine #16582

ivaylo-matov · 2025-10-08T11:05:40Z

Purpose

Draft PR — for review (please do not merge)

@mjkkirschner @twastvedt @zeusongit
Could you please review whether the approach is correct. The changes assume we’re still shipping DSPythonNet3 as a built-in package (same model as TuneUp) and that a future NuGet delivery will not require significant setup changes.

Problem:
Currently Dynamo uses the in-repo DSCPython project and the DLLs under \extern\Python, which are copied to \bin when Dynamo is built. By adding \extern\PythonNet3Engine (copied to \bin\Built-In Packages\packages\PythonNet3Engine), we end up with two Python.Runtime.dll files. Even if they target the same Python version, having two assemblies with the same identity on the probing path is fragile. We should use a single source of truth — the one that ships with DSPythonNet3.

What I’ve done:

src logic: Implemented new functionality (identical to DYN-9388 Make PythonNet3 default python engine as built-in package #16548)
Tests: Pointed tests to DSPythonNet3 methods where applicable
Extern cleanup:

removed the entire \extern\Python folder so we have a single source of truth from \extern\PythonNet3Engine

PythonMigrationViewExtension:

removed DSCPython reference.
switched Python.Runtime to the DSPythonNet3 version (build uses the copy under \extern\PythonNet3Engine\extra; at runtime Dynamo loads the copy under \bin\Built-In Packages\packages\PythonNet3Engine\extra).
minor ScriptMigrator updates for Python 3.11 behavior.

DynamoPythonTests:

removed DSCPython reference
added DSPythonNet3 reference
aligned Python.Runtime to the DSPythonNet3 version (same compile-time/runtime pattern as above)

Test logic changes:

CodeCompletionTests: now explicitly loads DSPythonNet3 in Setup()
PythonEvalTests: added [OneTimeSetUp] to load DSPythonNet3 and fetch DSPythonNet3Evaluator
PythonDependenciesTests: replaced the PythonNet3 package with the IronPython package

Solution:

removed DSCPython from the solution

Current status:
Most Python tests pass with this approach; a few are still failing (I’ll tale a look those now). Please confirm if:

using only the DSPythonNet3 Python.Runtime.dll is acceptable (single source of truth).
the compile-time vs runtime reference pattern above is OK for now (build from extern, load from Built-In Packages).

Thanks!

Declarations

Check these if you believe they are true

Is documented according to the standards
The level of testing this PR includes is appropriate
Changes to the API follow Semantic Versioning and are documented in the API Changes document.

Thanks!

Release Notes

Draft Pr to review if current strategy and test setup is correct and pparopriate.

Reviewers

@zeusongit

FYIs

@dnenov

This reverts commit 0389294.

…kage-does-not-load

cpython still present but does not load

…wip)

…com/ivaylo-matov/Dynamo into DYN-9388-Net3-package-does-not-load

CPythonClassCanBeReturnedAndSafelyDisposedInDownStreamNode

testing stuff CPythonClassCanBeReturnedAndSafelyDisposedInDownStreamNode few test fixed

…ithub.com/ivaylo-matov/Dynamo into DYN-9388-Net3-package-does-not-load-TESTS

github-actions

See the ticket for this pull request: https://jira.autodesk.com/browse/DYN-9388

Copilot

Pull Request Overview

This PR introduces a new PythonNet3 engine to replace the legacy CPython3 engine in Dynamo, transitioning to DSPythonNet3 as the built-in Python engine. The change removes the entire DSCPython project and updates all test files and engine references to use PythonNet3 instead of CPython3. The new engine is distributed as a built-in package similar to TuneUp, with a migration system that automatically converts CPython nodes and notifies users about the engine change.

Key Changes:

Replaced CPython3 engine with PythonNet3 engine throughout the codebase
Removed DSCPython project and dependencies from solution files
Added migration notifications and automatic conversion of legacy CPython nodes

Reviewed Changes

Copilot reviewed 96 out of 138 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/*/.dyn	Updated Python engine references from "CPython3" to "PythonNet3"
test/Libraries/DynamoPythonTests/*.cs	Updated test code to use DSPythonNet3 APIs and engine names
test/DynamoCoreTests/*.cs	Removed DSCPython dependencies and updated engine references
src/Libraries/DSCPython/*	Complete removal of DSCPython project files
src/PythonMigrationViewExtension/*	Updated to handle CPython to PythonNet3 migration
src/Libraries/PythonNodeModels*	Updated default engine and added migration UI elements
src/DynamoCoreWpf/*	Added UI support for engine change notifications
src/DynamoCore.sln	Removed DSCPython project references
extern/PythonNet3Engine/*	Added new built-in package files

Files not reviewed (4)

src/DynamoCoreWpf/Properties/Resources.Designer.cs: Language not supported
src/Libraries/DSCPython/Properties/Resources.Designer.cs: Language not supported
src/Libraries/PythonNodeModels/Properties/Resources.Designer.cs: Language not supported
src/PythonMigrationViewExtension/Properties/Resources.Designer.cs: Language not supported

Comments suppressed due to low confidence (5)

mjkkirschner · 2025-10-09T04:19:12Z

extern/PythonNet3Engine/extra/DSPythonNet3.deps.json

@@ -0,0 +1,317 @@
+{
+  "runtimeTarget": {


@twastvedt are these deps.json files part of the package? Do you know if their inclusion actually makes a difference to how the package is loaded?

deps.json files seem to be a standard part of .NET core output now. Everything I turn up online says they should be included, though there's little MS documentation that I can find. But I haven't done any tests to see what happens if you leave them out. I'd suggest they should be included unless we find problems with having them in.

mjkkirschner · 2025-10-09T04:19:52Z

extern/PythonNet3Engine/extra/DSPythonNet3Extension.xml

@@ -0,0 +1,15 @@
+<?xml version="1.0"?>


this does not seem valuable to include.

will remove in #16548

mjkkirschner

I don't think the changes you have made to the migrator / tests csproj files are the best way forward. I'd like to understand what you were seeing that made you think referencing the binary from the bin folder was required?

Just looking at it without building anything myself ;) - I would:

reference the binary from extern.
make sure the binary does not get copied during the build to the dynamo output path - because it's already going to be copied by the copy step that is added to copy the entire package.

Also - we need to move that copy step to a project that is built during a core only build.

Last I don't understand the changes that have been made to the test package - but I don't really remember what was in there in the first place.

mjkkirschner · 2025-10-09T04:24:57Z

src/DynamoCore/Configuration/PreferenceSettings.cs

        public bool ShowTabsAndSpacesInScriptEditor { get; set; }

+        /// <summary>
+        /// This defines if user wants to see the CPython3 update prompt.


I feel like this needs a super specific name or comment - because I could easily get this confused with our previous migration to Cpython3 from Ironpython2.

mjkkirschner · 2025-10-09T04:26:01Z

src/DynamoCore/Graph/Workspaces/WorkspaceModel.cs

+        [JsonIgnore]
+        public bool ShowCPythonNotifications
+        {
+            get => enableCPythonNotifications;


kind of weird pattern? Why not just make this an auto property?

mjkkirschner · 2025-10-09T04:26:39Z

src/DynamoCoreWpf/DynamoCoreWpf.csproj

 <Target Name="AfterBuildOps" AfterTargets="Build">
    <ItemGroup>
      <ExternTuneUpPkg Include="$(SolutionDir)..\extern\TuneUp\**\*.*" />
+      <ExternPythonNet3Pkg Include="$(SolutionDir)..\extern\PythonNet3Engine\**\*.*" />


this copy step should not be part of this wpf project, it should be party of Dynamo Core csproj -or some other csproj that is guaranteed to be part of a core only build so that even the DynamoCore non WPF sln will actually copy the files.

understood, dlls from \extern are not needed. DynamoCore copies the files from the NuGet now

mjkkirschner · 2025-10-09T04:27:45Z

src/DynamoCoreWpf/UI/GuidedTour/GuidesManager.cs

        /// </summary>
        /// <param name="content">The target content to display.</param>
        /// <param name="stayOpen">boolean indicates if the popup will stay open until user dismiss it.</param>
        /// TODO: Make this API out of guide manager to a more generic place


update the summary / params.

thank you. will push in #16548

mjkkirschner · 2025-10-09T04:28:32Z

src/DynamoCoreWpf/UI/Prompts/DynamoMessageBox.xaml

                                           Style="{StaticResource CustomRichTextBoxStyle}">
                </localui:CustomRichTextBox>

+                <!--  Checkbox  -->


does this effect other dynamo message box dialogs?

no, the checkbox is collapsed by default. Only one call turns it visible.

mjkkirschner · 2025-10-09T04:50:04Z

src/PythonMigrationViewExtension/PythonMigrationViewExtension.cs

+
+                    CurrentWorkspace.UndoRecorder.RecordModificationForUndo(node);
+                    node.EngineName = PythonEngineManager.PythonNet3EngineName;
+                    node.OnNodeModified();


so upgrading nodes will also run them?

mjkkirschner · 2025-10-09T04:53:53Z

src/PythonMigrationViewExtension/PythonMigrationViewExtension.csproj

+    </PropertyGroup>    
+    <ItemGroup>
+      <Reference Include="Python.Runtime">
+        <HintPath>..\..\bin\AnyCPU\$(Configuration)\Built-In Packages\packages\PythonNet3Engine\extra\Python.Runtime.dll</HintPath>


no, this does not look correct. This project cannot rely on a reference to a file in the bin folder that won't exist until after a build.

Reference the binary in the one in the extern folder and mark it private false so it does not get copied to the dynamo output path.
If it's getting copied anyway, then you need to inspect the debug output to see why.

Why do you think you need to reference this binary from this location?

mjkkirschner · 2025-10-09T04:56:42Z

test/Libraries/DynamoPythonTests/DynamoPythonTests.csproj

-      <HintPath>..\..\..\extern\Python\Python.Runtime.dll</HintPath>
-      <SpecificVersion>False</SpecificVersion>
+    <Reference Include="DSPythonNet3">
+      <HintPath>..\..\..\bin\AnyCPU\$(Configuration)\Built-In Packages\packages\PythonNet3Engine\extra\DSPythonNet3.dll</HintPath>


same here - this is not going to work / be reliable.

mjkkirschner · 2025-10-09T04:57:21Z

test/core/packageDependencyTests/PythonEnginePackage/extra/DSIronPython.runtimeconfig.json

@@ -0,0 +1,14 @@
+{


I think you should revert this file.

mjkkirschner · 2025-10-09T04:59:03Z

...packageDependencyTests/PythonEnginePackage/extra/IronPythonExtension_ExtensionDefinition.xml

@@ -0,0 +1,4 @@
+<ExtensionDefinition>


why have you added a copy of the ironPython package to the tests, did it already exist here?

ivaylo-matov · 2025-10-12T08:24:13Z

I don't think the changes you have made to the migrator / tests csproj files are the best way forward. I'd like to understand what you were seeing that made you think referencing the binary from the bin folder was required?

Just looking at it without building anything myself ;) - I would:

reference the binary from extern.

make sure the binary does not get copied during the build to the dynamo output path - because it's already going to be copied by the copy step that is added to copy the entire package.

Also - we need to move that copy step to a project that is built during a core only build.

Last I don't understand the changes that have been made to the test package - but I don't really remember what was in there in the first place.

@mjkkirschner thanks for the comments - super helpful.

I originally referenced the binaries from \bin because at the time I thought those were the ones Dynamo would use at runtime. I see now that was the wrong approach for compile-time references; \extern is the right source.

@zeusongit has provided a NuGet for the PythonNet3 engine. With that in place we no longer need the \extern\Python or \extern\PythonNet3Engine folders. I’ll add the NuGet to DynamoCore so the package is copied into the Built-In Packages folder at build time. For the migrator I’ll reference Python.Runtime via NuGet (compile-time only) and keep it engine-agnostic. In DynamoPythonTests I’ll reference the engine NuGet as well; at runtime the tests will load DSPythonNet3 from the Built-In Packages folder via a small one-time resolver. Please correct me if any of that sounds wrong.

Regarding the test package: PackageDependencyTests need a second package we can load to flip a few Python nodes to a different engine and verify behavior. When CPython was the default, we used the PythonNet3 package for that role. Now that PythonNet3 is the default, I need to replace that test package with something else. I’ve reviewed the new files I added for this purpose and will remove anything unnecessary.

Thanks again for the review. I’ll respond to a few comments here, and now that the way forward is clear to me I’ll address all your points and push the updates to #16548. Hope that's okay.

ivaylo-matov and others added 26 commits September 3, 2025 09:42

new DSPrythonNet3 project and PythonServices modified

0389294

Revert "new DSPrythonNet3 project and PythonServices modified"

b70f95d

This reverts commit 0389294.

Merge remote-tracking branch 'upstream/master' into DYN-9388-Net3-pac…

acb9b38

…kage-does-not-load

pythonnet3 loads as default

ee79b74

cpython still present but does not load

resources

62af16c

notifications transfer

cdabfa3

resources correction

2dfc883

cancel button removed

c0f5720

resources updates to match new design

250ad77

python update bar relocated

5c32588

request to update banner removed and new design for engine dropdown (…

68741ed

…wip)

python update bar relocated & request to update bar removed

95c1660

Merge branch 'DYN-9388-Net3-package-does-not-load' of https://github.…

a076326

…com/ivaylo-matov/Dynamo into DYN-9388-Net3-package-does-not-load

for pr

0058c33

public api register

e5f81c1

Update check_file_version.ps1

6434f92

Update check_file_version.ps1

7319898

Merge branch 'master' into DYN-9388-Net3-package-does-not-load

4b9a297

testing stuff

91202a8

CPythonClassCanBeReturnedAndSafelyDisposedInDownStreamNode

test fixed

90ee1bd

testing stuff CPythonClassCanBeReturnedAndSafelyDisposedInDownStreamNode few test fixed

PythonEvalTests

c5ba764

Merge branch 'DYN-9388-Net3-package-does-not-load-TESTS' of https://g…

4c04e35

…ithub.com/ivaylo-matov/Dynamo into DYN-9388-Net3-package-does-not-load-TESTS

references updated

58e9d88

DSCpython removed

78fc354

all python references pointing at built-in packages

2b2bba4

cleanup

7eeaf48

Copilot AI review requested due to automatic review settings October 8, 2025 11:05

github-actions bot reviewed Oct 8, 2025

View reviewed changes

Copilot AI reviewed Oct 8, 2025

View reviewed changes

zeusongit requested a review from mjkkirschner October 8, 2025 17:46

zeusongit requested review from a team and twastvedt October 8, 2025 17:46

mjkkirschner reviewed Oct 9, 2025

View reviewed changes

mjkkirschner requested changes Oct 9, 2025

View reviewed changes

ivaylo-matov added a commit to ivaylo-matov/Dynamo that referenced this pull request Oct 12, 2025

comments in DynamoDS#16582 addressed

c676f4b

zeusongit closed this Oct 15, 2025

DYN-9388: Draft - New PythonNet3 engine #16582

DYN-9388: Draft - New PythonNet3 engine #16582

Uh oh!

Conversation

ivaylo-matov commented Oct 8, 2025

Purpose

Declarations

Release Notes

Reviewers

FYIs

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes:

Reviewed Changes

Uh oh!

mjkkirschner Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjkkirschner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivaylo-matov commented Oct 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mjkkirschner Oct 9, 2025 •

edited

Loading

mjkkirschner left a comment •

edited

Loading