@MAPJe71 Thank you for your contribution.
My concern is the change of xml parser could bring some regression.
Therefore the unint test (with all source code files for every existing parser) is needed for validate the xml parser. Once we have unit test system for parsers we can integrate any modification.

Have you any idea for building such system ?

Hi @donho ,
The "left and right quotes" update to the XML parser applies to the dedicated "Function List" parser only. It makes sure both quotes surrounding a displayName- or an id-attribute are of the same "type" i.e. both single quotes or both double quotes. I updated the comment above.
Allthough (in retrospect) this "test"/"pre-condition" is more suitable to be part of the functionList.xml file validation.

I've been contemplating about unit tests myself.
Implementing new parsers is time consuming (or at least the way I do it) especially when they result in changes to existing parsers while preventing regressions. I've collected a set of "test" source files for about 200 different "languages" to help me with that (no I don't have a parser for every of these languages 😄 ). Implementing the functionList.xsd was actually the first step to get some test environment going.

My idea for Function List automated tests involves the Test Anything Protocol and Lua scripts using the standalone interpreter and/or the LuaScript plugin.
A test could consist of e.g.:

1. loading a source file in Notepad++, have Function List parse and "dump" the list, compare the dumped list with the expected list for that source file;
2. having a Lua script generate the source in the editor, "read" the list and compare it with the expected list for the generated code.

The expected list needs to be "connected" to its source file, these could be separate files but I prefer one file as not to loose their connection e.g. as comment in the source file (for example 1) or as part of the script (for example 2).

Do you already use some test system for (other parts of) Notepad++ or have any ideas?

It seems that it could take some time until all these function list parsers have been tested. I wrote a parser for Pascal/Delphi and I would like to share it with the community.

What would be better?

1. Wait until the above changes are integrated into a future release of Npp.
2. Fork the repository of @MAPJe71 and make a PR with my changes to it (I'm new to GitHub and Git, maybe the workflow is different).
3. Fork the Npp repository and make a PR to it.

@MAPJe71 The unit test system I use in my job (forget its name) needs to export all methods to test, each method will be compiled to a exe. They'll be launched after compiling to generate the result for comparing.
Such system doesn't work for Function list since all the parsers (regExprs) need Scintilla to evaluate the result.
I would say run compiled Notepad++ with divers source files to compare the function string output with the expect result, but I don't know yet how.
If you have any idea, please let me know.

@MAPJe71 I'm trying to make function list unit test happen, here is the first draft of interface:

notepad++.exe -funcListTest=L_CPP,msgboxOn,errLogOn,c:\funcListTests\cppTest.result c:\funcListTests\cppTest.cpp
if not ok show message box (opt) & write a log (opt) & exit (opt) return -1 
if ok exit return 0.

What do you think?

@donho
Questions / Remarks:

1. Why not have Notepad++ export the parse results and have the test logic (i.e. compare export with expected result) in a script or separate executable? Or is it required to have the test logic within Notepad++?
2. Do you have any objections on implementing the tests using scripts?
3. When required (see 1) will it only be included in the Debug build to not enlarge the Release build executable?
4. When having more tests implemented in the future for other parts of Notepad++ I would redefine the arguments in order not to "polute" the CLI to much e.g.

notepad++.exe -test=funcList,L_CPP,msgboxOn,errLogOn,c:\funcListTests\cppTest.result c:\funcListTests\cppTest.cpp

5. When not required (see 1) i.e. export the parse results, the command line interface (CLI) could be changed/simplified to

notepad++.exe -export=funcList c:\funcListTests\cppTest.cpp

6. To what extend/level is a test to validate a parser? Just the end result i.e. the tree as visual by a user, separate class parser and function parser results in a mixed parser, detecting comment sections, all of them?

ad.5. my preference 😄
ad.6. the comment sections part of the parser is not explicitly visible in the tree but I think we should be able to validate it separately.

@MAPJe71

Why not have Notepad++ export the parse results and have the test logic (i.e. compare export with expected result) in a script or separate executable?

By "export", do you mean write the strings into a file?

The approach that I suggested could be done without installation any script, that allows anyone test his/her RE very easily (for example, for user languages).
We do need scripting to execute every unit test anyway.

When having more tests implemented in the future for other parts of Notepad++ I would redefine the arguments in order not to "polute" the CLI to much e.g.
notepad++.exe -test=funcList,L_CPP,msgboxOn,errLogOn,c:\funcListTests\cppTest.result c:\funcListTests\cppTest.cpp

I'll consider this point.

When not required (see 1) i.e. export the parse results, the command line interface (CLI) could be changed/simplified to
notepad++.exe -export=funcList c:\funcListTests\cppTest.cpp

we need LangID still - the file extension could be used for several languages

@donho

By "export", do you mean write the strings into a file?

Yes, though I just realized we'd have to decide on a file format too e.g. XML (as already used within N++), JSON, YAML.

...anyone test his/her RE very easily (for example, for user languages).

They could also easily compare the exported file with an expected result file using any diff/compare tool. The files would preferably have the same fileformat though.

we need LangID still..

Good point, agreed.

We do need scripting to execute every unit test anyway.

Do you have a preference for a scripting language or maybe a preference which not to use?

@MAPJe71 Exported file could be JSON, however it'll be in the first implementation.
Here is the 2nd draft (I just remembered that we have -l flag).
Command:

notepad++.exe -export=functionList -lcpp c:\funcListTests\cppTest.cpp

the behaviour:

Notepad++ will open "c:\funcListTests\cppTest.cpp" as C++ source file, and write result to c:\funcListTests\cppTest.result - the old cppTest.result will be erased by the new one.

Do you have a preference for a scripting language or maybe a preference which not to use?

Not yet. What is your suggestion?

@donho

...the 2nd draft...

Wouldn't it make more sense to have the file extension of the result/export file reflect the used file format?
e.g.

c:\funcListTests\cppTestResult.json

or to also reflect the command line parameter

c:\funcListTests\cppTestExport.json

What is your suggestion?

The ones that come to mind are JavaScript, Lua, Perl, PowerShell, Python and VBScript. My first choice was Python, my second Lua. But then I remembered AutoHotkey and AutoIt. They support automated GUI testing out of the box which might be handy. Needs more research. But whatever the language I'd probably end up using some Command Shell scripts (i.e. batch files) too.

AutoHotkey

• Pros
  • Automated GUI testing
  • Unit Testing framework available
• Cons
  • Needs to be installed
  • Knowledge/support within community unknown
  • TAP producer available?
  • For MS Windows only

AutoIt

• Pros
  • Automated GUI testing
  • Unit Testing framework available (sort of)
• Cons
  • Needs to be installed
  • Knowledge/support within community unknown
  • TAP producer available?
  • For MS Windows only

JavaScript

• Pros
  • Interpreter natively available on Windows
  • Unit Testing framework available
  • TAP producer available
  • Notepad++ plugin available
• Cons
  • Knowledge/support within community unknown

KiXtart

• Pros
  • ?
• Cons
  • Needs to be installed
  • Knowledge/support within community unknown
  • TAP producer available?
  • For MS Windows only

Lua

• Pros
  • Basic interpreter has small footprint
  • Knowledge/support available within community
  • Unit Testing framework available
  • TAP producer available
  • Notepad++ plugin available
• Cons
  • Needs to be installed
  • Limited functionality of basic interpreter

Perl

• Pros
  • Unit Testing framework available
  • TAP producer available
• Cons
  • Needs to be installed e.g. ActivePerl
  • Steep learning curve
  • Knowledge/support within community unknown
  • Large footprint

PowerShell

• Pros
  • Interpreter natively available on Windows
  • Unit Testing framework available
  • TAP producer available
• Cons
  • Steep learning curve
  • Knowledge/support within community unknown
  • Only available on Windows?

Python

• Pros
  • Knowledge/support available within community
  • Unit Testing framework available
  • TAP producer available
  • Notepad++ plugin available
• Cons
  • Needs to be installed e.g. ActivePython
  • Large footprint

VBScript

• Pros
  • Interpreter natively available on Windows
  • Unit Testing framework available
• Cons
  • Knowledge/support within community unknown
  • Only available on Windows?

Edit: added KiXtart, AutoIt and AutoHotkey.

Wouldn't it make more sense to have the file extension of the result/export file reflect the used file format?

Indeed, it'll be:
c:\funcListTests\whatever.cpp.json

Thank you @MAPJe71 for these very detailled comparisons. I'll think about it.

@milipili I do remember that you told me that Unit test will be part of nany. What is its status? Is it usable immediately? (It's about comparing 2 result files, for the moment)

@MAPJe71
Please check the commit 1d58c1d in which fuc list explort is implemented (with no json).

Please let me know if there are somethings missing.

@MAPJe71
Let's go with Javascript - Would you like to take in charge of it?

@donho

Please let me know if there are somethings missing.

It appears that only the leaves of the tree are exported.

@MAPJe71 Please check the fix here: 78ea211

It would be nice if you also would include #3663, the missing Pascal/Delphi parser.

@MAPJe71 I'm exporting functionList in JSON.
Could you give me a Json for the following function list?

@dinkumoil
I will only update this PR for corrections/changes on already included parsers.

@donho

{
    "root":"lesDlgs.cpp",
    "branches":[
        {
            "name":"ValueDlg",
            "leaves":[
                "init",
                "doDialog",
                "reSizeValueBox",
                "run_dlgProc"
            ]
        },
        {
            "name":"ButtonDlg",
            "leaves":[
                "doDialog",
                "run_dlgProc"
            ]
        }
    ],
    "leaves":[
        "func1",
        "func2"
    ]
}

Thank you @MAPJe71
Wouldn't be better to remplace branches by nodes ?

@donho
When talking about a "tree" it might be better to replace root with trunk instead.
to-may-to - to-mah-to, I'll leave it up to you.

@MAPJe71 FYI (if you didn't chech the latest commits): function list export feature has been enhanced - in json format now.

For optimizations and regression checks an output of the parser timing could help to identify problematic parsers.

@MAPJe71 Just wanna know the progression of unit test for function list?
Are you blocked at some points? May I help ?

@donho
Hurdles:

1. It's not a JavaScript interpreter that is natively available on Windows but a JScript interpreter. Allthough JScript and JavaScript both are based on ECMAScript spec, they are of different generations, Microsoft JScript is based on an older spec and thus its interpreter can not handle the JavaScript scripts written with a newer spec in mind. E.g. JScript does not have a build-in "include/import/require" function. It is possible to define an "include/import/require" function to be able to use "library" scripts, but I think it will be a challenge for maintenance. We could use a standalone JavaScript interpreter (e.g. see list here) but this would mean a/another third party dependency for N++ (well at least for testing).
2. I get easily sidetracked :-D
3. Life happens

What do I have:

1. a basic concept of the test harness;
2. some batch scripts for the test harness (e.g. "createSandbox.cmd" to create an isolated test environment);
3. a JScript to validate the functionList.xml using available schema functionList.xsd;
4. a JScript "include/import/require" function;

Todo:

1. an "isEqual" function for JSON objects that does a "deep compare";
2. functionality to easily check/get/set N++ configuration options/settings for a test;
3. tests for the available programming languages;
4. schema files for N++ language/options/settings files: config.xml, langs.xml, stylers.xml, userDefineLang.xml, contextMenu.xml, localization*.xml;

@MAPJe71 Js in FunctionList has been broken since September 2016 due to this commit:
5d438aa

and maybe (since November 2016) due to:
fb189fa

UnitTest for FunctionList is needed définitively!

@donho @MAPJe71
What is the current status about the unit tests (which obviously need some documentation allowing people to check) and the updated functionlist?

@GitMensch Currently Notepad++ is capable to get input source file and output a result in JSON format.
All we need is have all input files in A folder and generate expected results in B folder, then in build system to integrate the ability to evoke notepad++ (in each build) to take files from A folder, and generate test files into C folder - this system should compare files between B & C folder.
I'll create an issue for it and I'll take in charge of this task. Any help for AppVeyor is welcome .

@donho that's very good news. Please drop a note when you're finished and I'll see how to add it to appveyor (other CI's are possible of course, too - but as long as this project is MS centered in its core it is likely best left there).

@MAPJe71
Sorry for the long await.
After the update of Uni-tests of functionlist being committed: 34357a4
please provide a new updated functionList.xml and the unit-test files if they are absent.
To avoid the regression, you should run unit-tests: make sure you copy your modified functionList.xml into C:\...\notepad-plus-plus\PowerEditor\bin, then open your powershell and go to C:\...\notepad-plus-plus\PowerEditor\Test\FunctionList to run .\unitTestLauncher.ps1. Once you see "All tests are passed." it's OK to do a PR :)