Create common test static lib and move UTs to TAEF#1849
Conversation
| @@ -0,0 +1,572 @@ | |||
| //****************************************************************************** | |||
There was a problem hiding this comment.
This file needs the most review.
|
Need to adjust the scripts still so that build definitions don't break. May be a bit non trivial since the output is a different format #Resolved |
|
Actually, we need 2 things:
I need to do the same things for getting functional tests running on hte build machine. In reply to: 275573399 [](ancestors = 275573399) |
Remove. I missed a few. The new outDir is /build/Release or /build/Debug. Perhaps we should do #Resolved #Resolved Refers to: build/Tests/UnitTests/CoreImage/CoreImage.UnitTests.vcxproj:73 in 53f7de8. [](commit_id = 53f7de8, deletion_comment = False) |
|
Also, sorry, I didn't think of this sooner.... In reply to: 275574435 [](ancestors = 275574435,275574352,275573399) |
| </ImportGroup> | ||
|
|
||
| <PropertyGroup> | ||
| <StripAppContainerBit>true</StripAppContainerBit> |
There was a problem hiding this comment.
StripAppContainerBit [](start = 5, length = 20)
:) #Closed
DHowett-MSFT
left a comment
There was a problem hiding this comment.
This is a partial review until I get in tomorrow.
| #include "EbrStorageFile.h" | ||
|
|
||
| #include <COMIncludes.h> | ||
| #include <wrl\client.h> |
There was a problem hiding this comment.
nit: standardize on /; we do that elsewhere in WOC code #Resolved
| <GenerateDebugInformation>true</GenerateDebugInformation> | ||
| <AdditionalDependencies>mincore.lib;libxml2.lib;icudt.lib;icuin.lib;icuuc.lib;libdispatch.lib;%(AdditionalDependencies)</AdditionalDependencies> | ||
| <AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;$(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> | ||
| <AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> |
There was a problem hiding this comment.
How did this ever work? #Resolved
There was a problem hiding this comment.
I mean I think it just ignores it if there is nothing there. WHich is true now. I forget what used to be there that CF needed. Scary indeed. #WontFix
| <ItemGroup> | ||
| <ClCompile Include="$(StarboardBasePath)\tests\unittests\Framework\Framework.cpp" /> | ||
|
|
||
| <ClangCompile Include="$(StarboardBasePath)\tests\unittests\CoreGraphics.Drawing\EntryPoint.cpp" /> |
There was a problem hiding this comment.
This guy uses a custom entry point so that it can take command line arguments. Is there going to be a way to pass in run-specific state with TAEF/TE?
It uses this state for:
- Generation mode (writing images to disk without testing)
- Comparison against a separate image root
There was a problem hiding this comment.
Yes but I was a little lazy. Its called RuntimeParameters. YOu can see it for the test directory stuff. We'd want to hide away those things behind a macro / function probably.
| test_info->test_case_name(), | ||
| test_info->name(), | ||
| CFSTR("TestImage.%s%s%@.png"), | ||
| GET_TEST_FULL_NAME().c_str(), |
There was a problem hiding this comment.
I cannot place a comment on the appropriate line since it's not in the diff.
DrawingTest uses RecordProperty to poke URL references out of the test output via XML.
A modified test runner can use these to display the images inline in its UI.
What's the TAEF equivalent? #Resolved
There was a problem hiding this comment.
I haven't looked through how we're using TAEF logging, but Log::File(L"path to file") would result in the image being saved under TAEF's current output directory.
From the TAEF docs;
Log:: File(const wchar_t* pszFileName)
Log a test file to be saved. Files are saved to a folder named "WexLogFileOutput". This folder will be in the folder specified by the WTTRunWorkingDir environment variable if that variable exists. When using Wex.Logger outside of TAEF and the WTTRunWorkingDir environment variable does not exist, the WexLogFileOutput folder will be in the current directory. When using Wex.Logger along with TAEF and the WTTRunWorkingDir environment variable does not exist, the WexLogFileOutput folder will be in the TAEF output folder.
If you just want to log the path to the file, you could Log::Property(const wchar_t* pszName, const wchar_t* pszValue). Of course, we'll need wttlog.dll to generate test logs.
From the TAEF docs;
Log::Property(const wchar_t* pszName, const wchar_t* pszValue)
Log a name/value property pair. The value can be in xml format.
In reply to: 98139951 [](ancestors = 98139951)
There was a problem hiding this comment.
Yeah sorry. I saw this but forgot to go make a macro for it. We'd want Log::Property I think. I'll make a macro for it. #Resolved
| #ifndef GET_TEST_FULL_NAME | ||
| #define GET_TEST_FULL_NAME \ | ||
| []() { \ | ||
| std::string alternateTestName = _ExchangeAlternateTestName(std::string(), false); \ |
There was a problem hiding this comment.
This scares me, but I'll have to look a little more at why.
There was a problem hiding this comment.
I agree. I was trying to avoid having to use a .cpp for this but I remembered that I already did that work for the logger so I could make this a bit nicer my having a class with more sane storage and helpers.
| #endif | ||
| testing::InitGoogleTest(&argc, argv); | ||
|
|
||
| _configImpl = std::move(std::make_shared<CommandLineDrawingTestConfigImpl>(argc, argv)); |
There was a problem hiding this comment.
Wait whoa no way, TE can pass args in argc/v?!
There was a problem hiding this comment.
no. it can't. this is me stamping out argc/v to interop with the below call. See my comment on runtime params.
| @@ -27,7 +28,8 @@ | |||
| #include "TestUtils.h" | |||
|
|
|||
| TEST(NSFileManager, GetAttributes) { | |||
| NSString* testFileFullPath = getPathToFile(@"/data/NSFileManagerUT.txt"); | |||
| NSString* testFileFullPath = | |||
| [[NSString stringWithUTF8String:GET_CURRENT_TEST_DIRECTORY().c_str()] stringByAppendingPathComponent:@"/data/NSFileManagerUT.txt"]; | |||
There was a problem hiding this comment.
I still like the idea of there being one nice shared thing that does this.. but for "lowest common denominator" support it would have to be either CFString or std::string/std::wstring returning. The CG.Drawing tests run as C++ only.
There was a problem hiding this comment.
I'm not sure what you mean... this is a std::string returning thing. The thing is just a bit weird. Are you wanting a more formalized class or something? I agree that the macro "function" isn't super clean ...
| <GenerateDebugInformation>true</GenerateDebugInformation> | ||
| <AdditionalDependencies>mincore.lib;libxml2.lib;icudt.lib;icuin.lib;icuuc.lib;libdispatch.lib;%(AdditionalDependencies)</AdditionalDependencies> | ||
| <AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;$(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> | ||
| <AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories> |
There was a problem hiding this comment.
yeah don't worry about it. There has been a lot of cleanup of the old code. You don't know half of the horrors of this project ;-) #WontFix
|
|
||
| std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> GetTestMetaFactories() { | ||
| std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> toReturn; | ||
| for (typename TestInfoContainer::iterator test_it = tests_.begin(); test_it != tests_.end(); ++test_it) { |
There was a problem hiding this comment.
nit: can make this range-based for? #Resolved
There was a problem hiding this comment.
There was a problem hiding this comment.
I'll try but I think I did that and it was complaining about needing the typename stuff. Maybe auto will work though.
| }; | ||
|
|
||
| // Get StorageFile reference for file in local folder | ||
| [WSStorageFile |
There was a problem hiding this comment.
I feel like I'm looking at node :(
There was a problem hiding this comment.
what do you mean? Is there something better I should be doing?
There was a problem hiding this comment.
No I don't think so, just oddly reminiscent of callbackapalooza
|
|
||
| TEST_CLASS_SETUP(AssetsLibraryClassSetup) { | ||
| bool success = AssetsLibraryTestVideoSetup("AssetsLibraryTestVideo.mp4"); | ||
| success &= FunctionalTestSetupUIApplication(); |
There was a problem hiding this comment.
Assuming we don't actually run the test if TEST_CLASS_SETUP fails, should we use the non-short circuiting &=? #WontFix
There was a problem hiding this comment.
I dunno this is copied from the current behavior in develop. I tried to not actually change the functional tests just consolidate them. But I may have messed up. You make a good point about if setup fails though to not run the test. I don't do that right in my test_p shim ... #WontFix
| // | ||
| // Cortana Activation Tests | ||
| // | ||
| class CortanaVoiceCommandForegroundActivation { |
There was a problem hiding this comment.
takes me back 😿 #Closed
| std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> GetTestMetaFactories() { | ||
| std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> toReturn; | ||
| for (typename TestInfoContainer::iterator test_it = tests_.begin(); test_it != tests_.end(); ++test_it) { | ||
| linked_ptr<TestInfo> test_info = *test_it; |
There was a problem hiding this comment.
linked_ptr [](start = 6, length = 20)
nit: linked_ptr& #Resolved
| // Pass null to indicate default app and app delegate classes | ||
| UIApplicationInitialize(nullptr, nullptr); | ||
| } | ||
|
|
There was a problem hiding this comment.
this shouldn't be resurrected; we now call UIApplicationInitialize in FunctionalTestHelpers::FunctionalTestSetupUIApplication() #Resolved
There was a problem hiding this comment.
yeah sorry bad merge on this file. I just wanted to delete everything in it basically but y'all made a bunch of changes to it. #Resolved
|
|
||
| class ActivatedAppReceivesToastNotificationTests { | ||
| BEGIN_TEST_CLASS(ActivatedAppReceivesToastNotificationTests) | ||
| END_TEST_CLASS() |
There was a problem hiding this comment.
nit: anywhere we're not adding class-level metadata, we can just use TEST_CLASS(ActivatedAppReceivesToastNotificationTests) (no need for begin/end) #WontFix
There was a problem hiding this comment.
I know but I'm not changing how it was for people already. This is just transplanted code. #lazy. #WontFix
| // 2. Use the same mechanism as in TEST_METHOD below to export a method from the WinObjC test file and call it here. | ||
| // 3. If you do not need this functionality feel free to remove this for your test class. | ||
| TEST_CLASS_CLEANUP(SampleTestClassCleanup) { | ||
| return true; |
There was a problem hiding this comment.
This should return FunctionalTestCleanupUIApplication #Resolved
| _EbrGetBaseWriteablePath | ||
|
|
||
| ; REMOVE / CLEANUP C++ EXPORTS | ||
| ?format@string@woc@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PBDZZ |
There was a problem hiding this comment.
probably? Despite outward appearances, my goal is not to fix everything potentially wrong with this project. #WontFix
There was a problem hiding this comment.
Although somewhat disappointing.
| } \ | ||
| catch (const ::WEX::TestExecution::VerifyFailureException&) \ | ||
| { \ | ||
| } \ |
There was a problem hiding this comment.
@DHowett-MSFT @jaredms this means that I should bail out and log a failure here right? Test Cleanup should still run though? Basically put the TestBody in the first try block?
There was a problem hiding this comment.
Yeah, I think so. gtest tries them all (?) in their own exception handlers i think.
| auto factory = metaFactory.second->CreateTestFactory(*param); \ | ||
| auto test = factory->CreateTest(); \ | ||
| \ | ||
| WEX::Logging::Log::StartGroup(currentTest.c_str()); \ |
There was a problem hiding this comment.
WEX::Logging::Log::StartGroup(currentTest.c_str()); \ [](start = 20, length = 53)
would be useful to also log what the parameters used here are. I think gtest has a method for that somewhere. I dunno if I can get to it off of the ParamGenerator though?
| size_t i = 0; \ | ||
| static std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; \ | ||
| static const std::wstring c_stringPrefix = L #prefix L"_" L#className L"." + converter.from_bytes(metaFactory.first.c_str()); \ | ||
| for (typename ::testing::internal::ParamGenerator<className::ParamType>::iterator param = paramGenerator.begin(); param != paramGenerator.end(); ++param, ++i) { \ |
There was a problem hiding this comment.
ypename ::testing::internal::ParamGeneratorclassName::ParamType::iterator [](start = 23, length = 76)
probably use auto& here?
| <# | ||
| .SYNOPSIS | ||
| Runs Tests for Islandwood project. | ||
| Runs Functional Tests for Islandwood project. |
There was a problem hiding this comment.
Can we just merge this with run-functionaltests? Would be nice to only "maintain" 1 script.
Granted, once these are written, they usually don't have to be modified.
|
This is all sorted out. In reply to: 275577758 [](ancestors = 275577758,275574435,275574352,275573399) |
7fe162c to
1a7732e
Compare
Looks like the test_p benchmark tests fail... Is it possible that you aren't using standard GTEST macros for that? Refers to: tests/Benchmark/EntryPoint.cpp:1 in a3246d1. [](commit_id = a3246d1, deletion_comment = False) |
| @@ -54,7 +48,22 @@ class CommandLineDrawingTestConfigImpl : public DrawingTestConfigImpl { | |||
|
|
|||
| public: | |||
| CommandLineDrawingTestConfigImpl(int argc, char** argv) | |||
There was a problem hiding this comment.
the TestConfig base class is pure virtual so that we don't need to clowncar all of the TAEF behaviours into the "command line" one ;P
There was a problem hiding this comment.
like: there are so many ifdefs in this file it might simply be easier to have a win32 file that has its own config impl and the os x file that keeps the commandline one.
There was a problem hiding this comment.
I agree in principle. I'll let you handle that. There were already a number of ifdefs in there for WIN32 so it seems like we need to think more about how to divide up the entrypoint code for the two.
| std::string tempBuffer; | ||
| uint32_t maxPath = _MAX_PATH; | ||
| tempBuffer.resize(maxPath); | ||
| _NSGetExecutablePath(&tempBuffer[0], &maxPath); |
There was a problem hiding this comment.
This introduces a dependency on Foundation, which not all of our test modules can take. We also cannot take that dependency on OS X.
There was a problem hiding this comment.
This is only for OSX and it is what was happening before anyway somewhere ... I didn't make up that code at least.
|
The only macros it uses are TEST_P and INSTANTIATE_TEST_CASE_P, but it still has the GTEST current_test_info()->name(), which is probably causing the crash. I'm building it now so I'll see if there are any other things that would cause an issue. Thanks for fixing everything else up! |
| __if_not_exists(GetTestFullName) { | ||
| static std::string GetTestFullName() { | ||
| std::string toReturn; | ||
| return toReturn + ::testing::UnitTest::GetInstance()->current_test_info()->test_case_name() + "." + |
There was a problem hiding this comment.
Why add toReturn+ here? It looks like it's being done for std::string coercion? That's not great.
There was a problem hiding this comment.
Yeah that is what it was for. I agree not great. I'll change.
|
I changed the test info name dealio and it seems to all pass for whatever that means... In reply to: 278493789 [](ancestors = 278493789) |
| *.xcf filter=lfs diff=lfs merge=lfs -text | ||
| *.gif filter=lfs diff=lfs merge=lfs -text | ||
| *.ttf filter=lfs diff=lfs merge=lfs -text | ||
| *.nupkg filter=lfs diff=lfs merge=lfs -text |
There was a problem hiding this comment.
Is this intended to be part of this change or part of the packaging change?
There was a problem hiding this comment.
This change does bring in the TAEF nupkg.
| const std::wstring& _IwGetWritableFolder() { | ||
| return g_WritableFolder; | ||
| unsigned int rawLength; | ||
| return WindowsGetStringRawBuffer(basePath.Get(), &rawLength); |
There was a problem hiding this comment.
length is marked _Out_opt_ in the header; don't make it populate something you're discarding immediately.
|
|
||
| // Set our writable and temp folders | ||
| IwSetWritableFolder(Windows::Storage::ApplicationData::Current->LocalFolder->Path->Data()); | ||
| // Set our temp folders |
There was a problem hiding this comment.
with the single line code, the comment is not necessary anymore (in the past, this used to be several lines :))
|
|
||
| #include <tuple> | ||
|
|
||
| // Copying these macros from WexTestClass.h, but not including |
There was a problem hiding this comment.
Based on this comment, the following macros could simply be destroyed? Or does this exist for cross-platform support?
There was a problem hiding this comment.
I didn't look at that. I assume those run cross-plat so i think we should keep as is.
| SB_EXPORT int EbrGetWantedOrientation(); | ||
|
|
||
| SB_EXPORT const wchar_t* IwGetWritableFolder(); | ||
| SB_EXPORT void IwSetWritableFolder(const wchar_t* folder); |
There was a problem hiding this comment.
I'd personally prefer to see all the Iw Writable Folder changes in a separate pull request with a separate commit (instead of squashed all together at the end). It will help future bisecting if we regress something.
In fact, I think all framework changes should be separate from this testing-related PR.
|
|
||
| // TODO:: | ||
| // todo-nithishm-04182016 - Investigate a better way to support ASSERT_*_MSG and EXPECT_*_MSG macros without re-defining them | ||
| // here. Bug 7244185 is tracking this work. |
There was a problem hiding this comment.
Should we port this VSO item to a GitHub item?
There was a problem hiding this comment.
Just remove the TODO. If we didn't need it after so long, we don't really need it :)
| #define FRIEND_TEST(test_case_name, test_name) friend class test_case_name##_##test_name | ||
|
|
||
| // clang-format off | ||
| template <unsigned NumChar> |
There was a problem hiding this comment.
A comment explaining this and why we need it would be excellent.
| str1[7] == 'D') ? L"TRUE" : L"FALSE"; | ||
| } | ||
|
|
||
| __declspec(noinline) inline std::string _ExchangeAlternateTestName( |
There was a problem hiding this comment.
extra special inlining. It means __declspec(selectany) for inline functions.
https://blogs.msdn.microsoft.com/freik/2005/10/26/how-to-make-a-no-inline-inline-function-and-why-you-might-want-to/ has a good summary.
The gist of it is that I'm using a global variable that needs to be shared across TUs (DrawingTest and the Test file that derives from it). This means I can't have more than one static function stamped out / use an inline function because the storage for the alternate name wouldn't be the same across TUs. However, I wanted to keep this in the header to minimize the additional goop needed by the tests (an alternate solution would be to place this stuff in the logger static lib). This means I need a "selectAny" type of behavior which you get from this hilarious syntax.
| } \ | ||
| catch (const ::WEX::TestExecution::VerifyFailureException&) \ | ||
| { \ | ||
| } \ |
There was a problem hiding this comment.
Yeah, I think so. gtest tries them all (?) in their own exception handlers i think.
| { \ | ||
| static std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; \ | ||
| std::wstring wideKey = converter.from_bytes(key); \ | ||
| std::wstring wideValue = converter.from_bytes(value); \ |
There was a problem hiding this comment.
Does Log::Property requiretypeof(value) == std::wstring? If not, and it's overloaded, we should pass value through unharmed for the appropriate implicit conversions to happen. This is too narrowly-specified as-is, it will only work for utf8 strings.
There was a problem hiding this comment.
I believe it requires wchar_t*s
| const char* testName = _STRINGIFY(__##test_class); \ | ||
| auto fullName = std::string(::testing::UnitTest::GetInstance()->current_test_info()->name()) + testName; \ | ||
| ::benchmark::internal::__RunCase(*m_test, fullName.c_str()); \ | ||
| ::benchmark::internal::__RunCase(*m_test, GetTestFullName().c_str()); \ |
There was a problem hiding this comment.
GetTestFullName() [](start = 46, length = 17)
I'll let you figure that out later. I don't think you do / you may want to rethink that.
There was a problem hiding this comment.
👍 TestFullName is the value this needs, you're right
|
(please file issues for anything you're marking as "you can deal with this later"; otherwise, we will all collectively lose track of them until the one day where we're like "hey wait this bug seems familiar ...") |
|
yep. Currently planning to file one on DrawingTests to integrate with the change (seems like we need to validate the arguments, perhaps adjust the ifdefs, and rework the test runner) and one for Benchmark tests In reply to: 278498755 [](ancestors = 278498755) |
| *.xcf filter=lfs diff=lfs merge=lfs -text | ||
| *.gif filter=lfs diff=lfs merge=lfs -text | ||
| *.ttf filter=lfs diff=lfs merge=lfs -text | ||
| *.nupkg filter=lfs diff=lfs merge=lfs -text |
There was a problem hiding this comment.
This change does bring in the TAEF nupkg.
|
|
||
| // Set our writable and temp folders | ||
| IwSetWritableFolder(Windows::Storage::ApplicationData::Current->LocalFolder->Path->Data()); | ||
| // Set our temp folders |
There was a problem hiding this comment.
with the single line code, the comment is not necessary anymore (in the past, this used to be several lines :))
| using namespace Microsoft::WRL; | ||
| using namespace Windows::Foundation; | ||
|
|
||
| Wrappers::HString GetAppDataPath() { |
There was a problem hiding this comment.
_IwGetWritablePath() won't work here?
There was a problem hiding this comment.
It will but I didn't really want to introduce that dependency to Starboard since its a weird private function anyway. I can though if we feel thats nicer? #Pending
There was a problem hiding this comment.
Your choice, it was a nit from my side.
| # Decide where the WTL output files will live | ||
| if ($WTLOutputDirectory -ne [string]$null) | ||
| { | ||
| $WTTLogFullPath = Join-Path $WTTLogPath wttlog.dll |
There was a problem hiding this comment.
Where would one get wttlog.dll?
There was a problem hiding this comment.
from a valid location. We can take this offline. #Closed
| ExecTest $argList | ||
|
|
||
| exit $failureCount + $crashingTestArray.length | ||
| exit |
There was a problem hiding this comment.
https://winobjc.visualstudio.com/WinObjC/_build/explorer?_a=summary&definitionType=2&buildId=7825
fails. It has some errors from CG.
In reply to: 100258200 [](ancestors = 100258200)
|
|
DHowett-MSFT
left a comment
There was a problem hiding this comment.
I'd still like to see the non-testing changes moved out into another pull request. I want that so much that I'm even willing to do the work necessary to tease them apart. 😦
e2bbce9 to
eba6f61
Compare
eba6f61 to
9bdbefd
Compare
a5f35bf to
4940d88
Compare
|
Woot |
No description provided.