Node markdown generator tool #11725

SHKnudsen · 2021-06-02T12:54:52Z

Purpose

This pr adds a new NodeDocumentationGenerator tool to automatically generate node documentation markdown files, the tool creates markdown files which (if specified) will have content from the DynamoDictionary added. The tool has two different verbs one that creates documentation from dlls in a directory, this can be used to create docs for Dynamo Core and Dynamo hosts, and another that generates docs from a package.

The PR also adds new APIs to ProtoCore.ProtoFFI.CLRDLLModule to enable scanning dlls in reflection mode, along with this new reflection extensions have been added to ProtoCore.Reflection.

The tool does not require Dynamo to be running but it uses the same APIs to scan dlls, main difference is that it dosent load anything into Dynamo but simply gives us back a list of all node types in the assembly being scanned

Dynamo loads documentation from Core and hosts by looking in a fallback_docs folder in the root directory of core or the host (TBD). By default when help is pressed on a node the DocumentationBrowser will look for the help doc associated with that node in the PackageDocumentationManager if no match is found it will first try and find a match in the host fallback_docs and secondly in Cores fallback_docs.

Besides that i made a slight modification to how the PackageDocumentationManager loads doc files, essentially i came across an issue where two packages was using the same namespace, this will cause an error on the manager. To address this i added the package name to the key of nodeDocumentationFileLookup.

Still need to add this to Dynamo.Mono, but lets touch base on that on our next call @mjkkirschner

Verbs

Verb	Description
`fromdirectory`	Generates documentation for all dlls (or only those specified in the filter) and all dyfs (if specified by the `--includedyfs` flag) in the input directory
`frompackage`	Looks up node libraries in the pkg.json file and creates documentation for those dlls and creates documentation for all dyfs in the `/dyf` folder. All generated files gets saved in `path/to/package/docs`

fromdirectory - flags

Short name	Long name	Optional	Description
`-i`	`--input`		Input binary file, containing nodes that documentaion should be generated for.
`-o`	`--output`		Location where markdown files should be generated
`-r`	`--references`	✅	optional flag - Folder paths to dlls that are used as references in the nodes
`-f`	`--filter`	✅	optional flag - Specifies which binary files documentation should be generated for
`-c`	`--includedyfs`	✅	optional flag - Include custom dyf nodes
`-d`	`--dictionary`	✅	optional flag - File path to DynamoDictionary json
`-w`	`--overwrite`	✅	optional flag - When specified the tool will overwrite files in the output path
`-y`	`--recursive-scan`	✅	optional flag - Input folder will be scanned recursively
`-s`	`--compress-images`	✅	optional flag - When set, the tool will try to compress images from dictionary content
`-x`	`--layout-spec`	✅	optional flag - Path to a LayoutSpecification json file

fromdirectory - samples

fromdirectory 
-i "C:\Users\....\Documents\GitHub\Dynamo\bin\AnyCPU\Debug\nodes" 
-f CoreNodeModels.dll 
-o "C:\Users\...\Desktop\TestMdOutput_CoreNodeModels" 
-d "C:\Users\...\Documents\GitHub\DynamoDictionary\public\data\Dynamo_Nodes_Documentation.json" 
-x "C:\Users\...\Documents\GitHub\Dynamo\src\LibraryViewExtension\web\library\layoutSpecs.json" 
-w -s

fromdirectory 
-i "C:\Users\...\Documents\GitHub\DynamoRevit\bin\AnyCPU\Release\Revit\nodes" 
-f DSRevitNodesUI.dll 
-o "C:\Users\...\Desktop\TestMdOutputRevitNodes - Compressed" 
-d "C:\Users\...\Documents\GitHub\DynamoDictionary\public\data\Dynamo_Nodes_Revit.json" 
-x "C:\Users\...\Documents\GitHub\DynamoRevit\src\DynamoRevit\Resources\LayoutSpecs.json" 
-r "C:\Users\...\Desktop\PathToFolderWithRevitAPI" "C:\...\DynamoRevit\bin\AnyCPU\Debug\Revit"
-w -s

frompackage - flags

Short name	Long name	Optional	Description
`-i`	`--input`		Package folder path.
`-r`	`--references`	✅	optional flag - Folder paths to dlls that are used as references in the nodes
`-w`	`--overwrite`	✅	optional flag - When specified the tool will overwrite files in the output path

frompackage - samples

frompackage 
-i "C:\Users\...\source\repos\InspectionTest\InspectionTest\dist\InspectionTest" 
-r "C:\Users\...\Desktop\PathToFolderWithRevitAPI"
-w

Docs in Revit

Declarations

Check these if you believe they are true

The codebase is in a better state after this PR
Is documented according to the standards
The level of testing this PR includes is appropriate
User facing strings, if any, are extracted into *.resx files
All tests pass using the self-service CI.
Snapshot of UI changes, if any.
Changes to the API follow Semantic Versioning and are documented in the API Changes document.
This PR modifies some build requirements and the readme is updated

Reviewers

@mjkkirschner

FYIs

@Amoursol

…erator-tool

mjkkirschner · 2021-06-07T19:35:11Z

@SHKnudsen - this PR looks to need work before review:

conflicts
based on an old commit that is not building.

src/DocumentationBrowserViewExtension/DocumentationBrowserViewModel.cs

src/DocumentationBrowserViewExtension/PackageDocumentationManager.cs

mjkkirschner · 2021-07-09T20:11:40Z

@SHKnudsen was trying to build this and ran into the issue that -l no longer exists for loggerpath, can you update the PR description? Also there is a conflict on this branch stopping the build from completing.

…erator-tool

SHKnudsen · 2021-07-11T15:37:19Z

@SHKnudsen was trying to build this and ran into the issue that -l no longer exists for loggerpath, can you update the PR description? Also there is a conflict on this branch stopping the build from completing.

@mjkkirschner updated the description and solved the merge conflict. Also made the changes that we discussed, and everything seems to work nicely now.

mjkkirschner · 2021-07-11T22:20:53Z

@SHKnudsen thats great to hear, I am currently looking into creating a custom importModuleHandler that will either:

ignore import statements
redirect import statements to your getFunctionDescriptorReflection methods.

I'll send a PR to your fork if that works for you?

mjkkirschner · 2021-07-11T22:21:52Z

@Amoursol can you please check in with legal again on the inclusion of imageMagik - with its custom license - we won't be able to finalize this without that approval - if needed we can back those changes out and add compression later I guess.

SHKnudsen · 2021-07-12T09:31:51Z

I'll send a PR to your fork if that works for you?

Yep that sounds good, thanks!

Amoursol · 2021-07-12T13:16:33Z

@Amoursol can you please check in with legal again on the inclusion of imageMagik - with its custom license - we won't be able to finalize this without that approval - if needed we can back those changes out and add compression later I guess.

Will chase on this now @mjkkirschner, may have slipped my mind. Sorry for that!

put clr into reflection mode when importing DS files fix classname for generated ds docs

fix issue with protogeometry format load error

SHKnudsen · 2021-07-19T08:43:12Z

@mjkkirschner merged your DS file changes so think this is getting close to being ready. I took out the tests on this PR so we can add everything test-related in a new PR.

aparajit-pratap

More comments to come ...

src/DocumentationBrowserViewExtension/PackageDocumentationManager.cs

src/DocumentationBrowserViewExtension/DocumentationBrowserViewExtension.cs

aparajit-pratap · 2021-08-09T19:29:45Z

src/DynamoCore/Core/NodeModelAssemblyLoader.cs

+        /// <param name="t">Type to check</param>
+        /// <param name="nodeModelType">Type of NodeModel, this is needed to be sure we are comparing the same NodeModel type reference</param>
+        /// <returns></returns>
+        internal static bool IsNodeSubTypeReflectionLoaded(Type t, Type nodeModelType)


What does the name of this method mean? Can we rename it to have a simpler meaning?

If this method is truly necessary, looks like there could be room to get rid of some code duplication between this and the IsNodeSubType method.

aparajit-pratap · 2021-08-09T19:35:06Z

src/DynamoCore/Graph/Nodes/TypeLoadData.cs

+            IsHidden = attributes.OfType<IsVisibleInDynamoLibraryAttribute>().Any(attr => !attr.Visible);
+
+            var attribs = attributes.OfType<NodeNameAttribute>();
+            if (attribs.Any() && !IsDeprecated && !IsMetaNode && IsDSCompatible && !IsHidden)


If an attribute is a NodeNameAttribute, how is there a possibility that any of the other flags, IsDeprecated, etc. will be true?

Not sure i fully understand the question?

The type can have both NodeNameAttribute and NodeDeprecatedAttribute and so on. I dont know the exact reason for making this check, but am simply following the logic of the original TypeLoadData constructor
https://github.com/DynamoDS/Dynamo/blob/master/src/DynamoCore/Graph/Nodes/TypeLoadData.cs#L43

appears to me those other flags are checked above by checking other attributes.

src/DynamoCore/Library/FunctionDescriptor.cs

aparajit-pratap · 2021-08-10T01:11:13Z

src/Engine/ProtoCore/FFI/CLRDLLModule.cs

+                if ((retype.IsIndexable && mattrs.AllowRankReduction)
+                    || (typeof(object).Equals(method.ReturnType)))
+                {
+                    retype.rank = Constants.kArbitraryRank;
+                }
+
+                func.Signature = ParseArgumentSignature(method);


Other than GetProtoCoreType and isPropertyAccessor is this code also expected to throw in reflection-only mode? If not, we could probably move this out of the try-catch?

I guess some of it might not be, but lots of its relying on the retype so if that fails i would say we dont want to execute this code either

src/Engine/ProtoCore/FFI/CLRDLLModule.cs

src/Engine/ProtoCore/FFI/FFIHandler.cs

aparajit-pratap · 2021-08-10T01:45:09Z

src/Engine/ProtoCore/Reflection/MemberInfoReflectionExtensions.cs

+            try
+            {
+                // if overriding object 
+                var baseType = methodInfo.DeclaringType.BaseType;


Shouldn't this method be recursive - base type of base type and so on until you find a matching methodInfo?

I just did a little test with 3 classes ParentType, MiddelType and NestedType
(inheritance like this NestedType -> MiddelType -> ParentType) where ParentType has a virtual RandomProp which NestedType implements. In that scenario doing GetBaseDefinitionReflectionContext on NestedType.RandomProp will return the correct methodInfo from ParentType

src/Engine/ProtoCore/Reflection/MemberInfoReflectionExtensions.cs

src/DynamoCore/Search/SearchElements/ZeroTouchSearchElement.cs

src/DynamoCLI/Program.cs

aparajit-pratap · 2021-08-10T20:31:34Z

src/Tools/NodeDocumentationMarkdownGenerator/AssemblyHandler.cs

+                    }
+                }
+
+                if (functionGroups.Count > 0)


if(functionGroups.Any()?

src/Tools/NodeDocumentationMarkdownGenerator/AssemblyHandler.cs

src/Tools/NodeDocumentationMarkdownGenerator/Commands/FromDirectoryCommand.cs

src/Tools/NodeDocumentationMarkdownGenerator/MarkdownHandler.cs

src/Tools/NodeDocumentationMarkdownGenerator/AssemblyHandler.cs

Co-authored-by: michael kirschner <michael.kirschner@autodesk.com>

add gif compression by quantization to 32k colors in debug leave console open until readline for easy log reading

* Dyn 3817 exit tour window popup (DynamoDS#11918) * DYN-3817-ExitTourWindow Popup I added the UI for the Exit Tour Window, but still having some issues in the layout * DYN-3817-ExitTourWindow-Popup I created the view for the Exit Tour Window and also the functionality of showing the ExitTour ( this is shown only when one of the steps is closed so the tour is not finished - this doesn't apply for the Survey popup). A resource string was added and is related to the ExitTour content so it can be localized later. * DYN-3817-ExitTourWindow-Popup Removing code not needed in DynamoView.xaml.cs * DYN-3817-ExitTourWindow-Popup Code Review The ExitTourWindow was renamed to RealTimeInfoWindow. I reorganized the RealTimeInfoWindow so doesn't depend of a Step (then it can be used in a more generic way). I created the properties Width, Height and TextContent inside the RealTimeInfoWindow so they can be set up from outside. The TextContent property contains the text that will be shown in the popup and it can be updated on runtime. * DYN-3817-ExitTourWindow-Popup Code Issues Fixing issues produced after merging the latest changes in master branch * DYN-3817-ExitTourWindow-Popup Code Review 2 The offset hardcoded values were changed to const double, also I reorganized the way the RealTimeInfoWindow object is created. Also I added some comments in the xaml file * Dyn-3640 - Recover execution after clearing cyclic dependency from graph (DynamoDS#11896) * cleanup * minor cleanup * clear cycles from dependency graph nodes for graph runs * cleanup * update failing test * clear node warnings for cleared cycles * do not include cyclic graph node as a redefined node of another graph node * add unit tests * fix one more case * review comments * add logging to intermittent pkgloader tests (DynamoDS#11925) * update tests * update xml summary add some logging to failing tests Co-authored-by: michael kirschner <michael.kirschner@autodesk.com> * pkg load test log typo (DynamoDS#11926) * update tests * update xml summary add some logging to failing tests * typo Co-authored-by: michael kirschner <michael.kirschner@autodesk.com> * Update UI (DynamoDS#11932) * Update searchMethod invocation in LibraryMSWebBrowserExtension (DynamoDS#11931) * Update searchMethod invocation Update searchMethod invocation to use correct number of parameters * Update SearchDictionary.cs * Add adsk fonts to extern folder with license terms. (DynamoDS#11890) * small adjustment * align titles * add names to buttons for ui automation tests * missing button name * new font folder for use in Dynamo only * Update readme.md * add font links to about box Co-authored-by: michael kirschner <michael.kirschner@autodesk.com> * check for null (DynamoDS#11930) * Prevent port measure crash (DynamoDS#11935) * CRASH: Delete node with right-click at the same time (DynamoDS#11927) * null checks * Add unit test * Fix For Loading Json File (DynamoDS#11934) * Fix For Loading Json File During the testing of the task https://jira.autodesk.com/browse/DYN-3817 Aabishkar reported that when trying to open the "Get Started" tour a message appear saying Could not find a part of the path 'C:\Windows\system32\UI\GuidedTour\dynamo_guides.json' Then I modified the code that is loading the json,file, in the past was a relative path and now we are using an absolute path. * Fix For Loading Json File Instead to have the json file path in the GuidesManager constructor I created two public properties that will contains the assembly path execution and the JSON files guides path, so we can verify in the unit testing that this paths exists and if the Guide will be created correctly. Also I added a unit test that will validate that the json guides file exists and that the Guides can be created. * Handle crash due to save path being too long Co-authored-by: Roberto T <61755417+RobertGlobant20@users.noreply.github.com> Co-authored-by: aparajit-pratap <aparajit.pratap@autodesk.com> Co-authored-by: michael kirschner <michael.kirschner@autodesk.com> Co-authored-by: Aaron (Qilong) <173288704@qq.com> Co-authored-by: Ashish Aggarwal <ashish.aggarwal@autodesk.com>

add new flag -g to compress gifs

SHKnudsen added 18 commits April 14, 2021 16:22

initial commit

f9093e4

Use CLRDLLModule

354e682

Refactoring

3dd5ad4

add hosts flag and other additions to frompackage

05f5984

🧹🧹

d7fab57

Update Program.cs

f3c2bd0

clean up + handle overloads

bf971d0

🧹

e5ea6c2

Update AssemblyHandler.cs

70bf7c3

too many things 🤯

67353b4

Save images instead of embedding

999a0b5

comment updates

f8c13eb

add support for .ds files

7548c79

comments + move to tools folder

97d29e4

add reference to RestSharp - need to create PackageUploadRequestBody

7599dfb

Update FromPackageFolderCommand.cs

f1f5df5

Merge remote-tracking branch 'upstream/master' into Node-Markdown-gen…

a7b1542

…erator-tool

🧹

cc9705e

Amoursol requested review from QilongTang and mjkkirschner June 2, 2021 13:19

SHKnudsen added 6 commits June 3, 2021 11:29

Remove logger

3cc3667

remove unused using

e372af6

Update FromDirectoryOptions.cs

62009d5

Update AssemblyHandler.cs

b3443ed

Change project type + add test project

f9a5627

remove old files

6772242

mjkkirschner reviewed Jun 7, 2021

View reviewed changes

src/DocumentationBrowserViewExtension/DocumentationBrowserViewModel.cs Outdated Show resolved Hide resolved

mjkkirschner reviewed Jun 7, 2021

View reviewed changes

src/DocumentationBrowserViewExtension/PackageDocumentationManager.cs Outdated Show resolved Hide resolved

mjkkirschner reviewed Jun 7, 2021

View reviewed changes

src/DocumentationBrowserViewExtension/PackageDocumentationManager.cs Show resolved Hide resolved

get baseType with overloads

a7bd10c

SHKnudsen added 2 commits July 11, 2021 16:28

Load dynamo dlls from main bin folder + fix overload lookup

b65a2fd

Merge remote-tracking branch 'upstream/master' into Node-Markdown-gen…

a0c1eb2

…erator-tool

Update DocumentationBrowserViewExtensionTests.cs

5559e8b

mjkkirschner and others added 3 commits July 13, 2021 15:08

fix issue with protogeometry format load error

1138c92

put clr into reflection mode when importing DS files fix classname for generated ds docs

remove unused

d9b74e1

Merge pull request #25 from mjkkirschner/nmgt_mjk

259f870

fix issue with protogeometry format load error

SHKnudsen and others added 2 commits July 19, 2021 09:44

remove test project

2b24e18

Merge branch 'master' into Node-Markdown-generator-tool

18aa5b9

aparajit-pratap reviewed Aug 10, 2021

View reviewed changes

aparajit-pratap reviewed Aug 11, 2021

View reviewed changes

comment cleanups

2cec9a2

pinzart90 reviewed Aug 16, 2021

View reviewed changes

src/Tools/NodeDocumentationMarkdownGenerator/AssemblyHandler.cs Show resolved Hide resolved

mjkkirschner and others added 8 commits August 16, 2021 13:42

merge license (#31)

0131c59

Co-authored-by: michael kirschner <michael.kirschner@autodesk.com>

move LayoutSpecification types to DynamoCore with type forward (#30)

c0c0632

Co-authored-by: michael kirschner <michael.kirschner@autodesk.com>

add new flag -g to compress gifs

c76af1c

add gif compression by quantization to 32k colors in debug leave console open until readline for easy log reading

typo

35ef80d

Merge branch 'master' into Node-Markdown-generator-tool

2a223fb

Merge pull request #32 from mjkkirschner/nmdg-imagecompgif

9d88dd2

add new flag -g to compress gifs

add dyf folder exists check

15cb6e6

mjkkirschner mentioned this pull request Aug 19, 2021

node docs generator tool without MetaDataLoadContext #11950

Merged

10 tasks

mjkkirschner closed this Oct 22, 2021

Node markdown generator tool #11725

Node markdown generator tool #11725

Uh oh!

Conversation

SHKnudsen commented Jun 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Verbs

fromdirectory - flags

fromdirectory - samples

frompackage - flags

frompackage - samples

Docs in Revit

Declarations

Reviewers

FYIs

Uh oh!

mjkkirschner commented Jun 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mjkkirschner commented Jul 9, 2021

Uh oh!

SHKnudsen commented Jul 11, 2021

Uh oh!

mjkkirschner commented Jul 11, 2021

Uh oh!

mjkkirschner commented Jul 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SHKnudsen commented Jul 12, 2021

Uh oh!

Amoursol commented Jul 12, 2021

Uh oh!

SHKnudsen commented Jul 19, 2021

Uh oh!

aparajit-pratap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SHKnudsen Aug 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

SHKnudsen commented Jun 2, 2021 •

edited

Loading

mjkkirschner commented Jul 11, 2021 •

edited

Loading

SHKnudsen Aug 12, 2021 •

edited

Loading