Skip to content
This repository was archived by the owner on May 28, 2025. It is now read-only.

Create common test static lib and move UTs to TAEF#1849

Merged
bbowman merged 3 commits into
microsoft:developfrom
bbowman:bbowman/updateTAEF+RedoUTs
Feb 10, 2017
Merged

Create common test static lib and move UTs to TAEF#1849
bbowman merged 3 commits into
microsoft:developfrom
bbowman:bbowman/updateTAEF+RedoUTs

Conversation

@bbowman

@bbowman bbowman commented Jan 27, 2017

Copy link
Copy Markdown
Member

No description provided.

@bbowman

bbowman commented Jan 27, 2017

Copy link
Copy Markdown
Member Author

#include <TestFramework.h>

I don't know what an .inl file is??? This looked like it needed to change though.


Refers to: tests/unittests/Starboard/StarboardAutoIdTests.inl:17 in 53f7de8. [](commit_id = 53f7de8, deletion_comment = False)

@@ -0,0 +1,572 @@
//******************************************************************************

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file needs the most review.

@bbowman

bbowman commented Jan 27, 2017

Copy link
Copy Markdown
Member Author

Need to adjust the scripts still so that build definitions don't break. May be a bit non trivial since the output is a different format #Resolved

@pradipd

pradipd commented Jan 27, 2017

Copy link
Copy Markdown
Contributor

Actually, we need 2 things:

  1. We need to get wttlog.dll from somewhere to create output files.
  2. We need a way to upload the results to VSO.

I need to do the same things for getting functional tests running on hte build machine.
So, please hold off merging before I figure that out. Hopefully I'll have it figured out tomorrow (we already do this in the lab).


In reply to: 275573399 [](ancestors = 275573399)

@pradipd

pradipd commented Jan 27, 2017

Copy link
Copy Markdown
Contributor

Also, #1 is harder than #2.


In reply to: 275574352 [](ancestors = 275574352,275573399)

@bbowman

bbowman commented Jan 27, 2017

Copy link
Copy Markdown
Member Author

EndProject

CoreGraphics, CoreImage, and COreGraphcis.Drawing had a different OutDir. This should be standardized. I need to figure out what I missed. #Resolved


Refers to: build/build.sln:90 in 53f7de8. [](commit_id = 53f7de8, deletion_comment = False)

@bbowman

bbowman commented Jan 27, 2017

Copy link
Copy Markdown
Member Author
<OutDir>$(SolutionDir)$(Platform)\$(Configuration)\$(RootNamespace)</OutDir>

Remove. I missed a few. The new outDir is /build/Release or /build/Debug. Perhaps we should do $(SolutionDir)$(Platform)$(Configuration)\ so its close to the same? It just needs to be different from the frameworks out dir so that the package .zip stuff (soon to be removed) doesn't pick up uts. This means we will have 1 copy of the binaries around to support the UTs but that seems a lot better.

#Resolved #Resolved


Refers to: build/Tests/UnitTests/CoreImage/CoreImage.UnitTests.vcxproj:73 in 53f7de8. [](commit_id = 53f7de8, deletion_comment = False)

@pradipd

pradipd commented Jan 27, 2017

Copy link
Copy Markdown
Contributor

Also, sorry, I didn't think of this sooner....


In reply to: 275574435 [](ancestors = 275574435,275574352,275573399)

</ImportGroup>

<PropertyGroup>
<StripAppContainerBit>true</StripAppContainerBit>

@pradipd pradipd Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StripAppContainerBit [](start = 5, length = 20)

:) #Closed

@DHowett-MSFT DHowett-MSFT left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a partial review until I get in tomorrow.

Comment thread Frameworks/Starboard/EbrFile.cpp Outdated
#include "EbrStorageFile.h"

#include <COMIncludes.h>
#include <wrl\client.h>

@DHowett-MSFT DHowett-MSFT Jan 27, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: standardize on /; we do that elsewhere in WOC code #Resolved

<GenerateDebugInformation>true</GenerateDebugInformation>
<AdditionalDependencies>mincore.lib;libxml2.lib;icudt.lib;icuin.lib;icuuc.lib;libdispatch.lib;%(AdditionalDependencies)</AdditionalDependencies>
<AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;$(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

@DHowett-MSFT DHowett-MSFT Jan 27, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol?!?! #Closed

@DHowett-MSFT DHowett-MSFT Jan 27, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did this ever work? #Resolved

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean I think it just ignores it if there is nothing there. WHich is true now. I forget what used to be there that CF needed. Scary indeed. #WontFix

<ItemGroup>
<ClCompile Include="$(StarboardBasePath)\tests\unittests\Framework\Framework.cpp" />

<ClangCompile Include="$(StarboardBasePath)\tests\unittests\CoreGraphics.Drawing\EntryPoint.cpp" />

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guy uses a custom entry point so that it can take command line arguments. Is there going to be a way to pass in run-specific state with TAEF/TE?

It uses this state for:

  • Generation mode (writing images to disk without testing)
  • Comparison against a separate image root

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but I was a little lazy. Its called RuntimeParameters. YOu can see it for the test directory stuff. We'd want to hide away those things behind a macro / function probably.

test_info->test_case_name(),
test_info->name(),
CFSTR("TestImage.%s%s%@.png"),
GET_TEST_FULL_NAME().c_str(),

@DHowett-MSFT DHowett-MSFT Jan 27, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot place a comment on the appropriate line since it's not in the diff.

DrawingTest uses RecordProperty to poke URL references out of the test output via XML.
A modified test runner can use these to display the images inline in its UI.

What's the TAEF equivalent? #Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked through how we're using TAEF logging, but Log::File(L"path to file") would result in the image being saved under TAEF's current output directory.

From the TAEF docs;

Log:: File(const wchar_t* pszFileName)
Log a test file to be saved. Files are saved to a folder named "WexLogFileOutput". This folder will be in the folder specified by the WTTRunWorkingDir environment variable if that variable exists. When using Wex.Logger outside of TAEF and the WTTRunWorkingDir environment variable does not exist, the WexLogFileOutput folder will be in the current directory. When using Wex.Logger along with TAEF and the WTTRunWorkingDir environment variable does not exist, the WexLogFileOutput folder will be in the TAEF output folder.

If you just want to log the path to the file, you could Log::Property(const wchar_t* pszName, const wchar_t* pszValue). Of course, we'll need wttlog.dll to generate test logs.

From the TAEF docs;

Log::Property(const wchar_t* pszName, const wchar_t* pszValue)
Log a name/value property pair. The value can be in xml format.


In reply to: 98139951 [](ancestors = 98139951)

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sorry. I saw this but forgot to go make a macro for it. We'd want Log::Property I think. I'll make a macro for it. #Resolved

Comment thread tests/Tests.Shared/test-api.h Outdated
#ifndef GET_TEST_FULL_NAME
#define GET_TEST_FULL_NAME \
[]() { \
std::string alternateTestName = _ExchangeAlternateTestName(std::string(), false); \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scares me, but I'll have to look a little more at why.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I was trying to avoid having to use a .cpp for this but I remembered that I already did that work for the logger so I could make this a bit nicer my having a class with more sane storage and helpers.

#endif
testing::InitGoogleTest(&argc, argv);

_configImpl = std::move(std::make_shared<CommandLineDrawingTestConfigImpl>(argc, argv));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait whoa no way, TE can pass args in argc/v?!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. it can't. this is me stamping out argc/v to interop with the below call. See my comment on runtime params.

@@ -27,7 +28,8 @@
#include "TestUtils.h"

TEST(NSFileManager, GetAttributes) {
NSString* testFileFullPath = getPathToFile(@"/data/NSFileManagerUT.txt");
NSString* testFileFullPath =
[[NSString stringWithUTF8String:GET_CURRENT_TEST_DIRECTORY().c_str()] stringByAppendingPathComponent:@"/data/NSFileManagerUT.txt"];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still like the idea of there being one nice shared thing that does this.. but for "lowest common denominator" support it would have to be either CFString or std::string/std::wstring returning. The CG.Drawing tests run as C++ only.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean... this is a std::string returning thing. The thing is just a bit weird. Are you wanting a more formalized class or something? I agree that the macro "function" isn't super clean ...

@aballway aballway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔑

<GenerateDebugInformation>true</GenerateDebugInformation>
<AdditionalDependencies>mincore.lib;libxml2.lib;icudt.lib;icuin.lib;icuuc.lib;libdispatch.lib;%(AdditionalDependencies)</AdditionalDependencies>
<AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;$(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
<AdditionalLibraryDirectories>$(StarboardBasePath)\Frameworks\limbo;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

@aballway aballway Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

limbo? #Closed

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah don't worry about it. There has been a lot of cleanup of the old code. You don't know half of the horrors of this project ;-) #WontFix


std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> GetTestMetaFactories() {
std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> toReturn;
for (typename TestInfoContainer::iterator test_it = tests_.begin(); test_it != tests_.end(); ++test_it) {

@aballway aballway Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can make this range-based for? #Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, also auto would be nice


In reply to: 98142566 [](ancestors = 98142566)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try but I think I did that and it was complaining about needing the typename stuff. Maybe auto will work though.

};

// Get StorageFile reference for file in local folder
[WSStorageFile

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like I'm looking at node :(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean? Is there something better I should be doing?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I don't think so, just oddly reminiscent of callbackapalooza


TEST_CLASS_SETUP(AssetsLibraryClassSetup) {
bool success = AssetsLibraryTestVideoSetup("AssetsLibraryTestVideo.mp4");
success &= FunctionalTestSetupUIApplication();

@aballway aballway Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming we don't actually run the test if TEST_CLASS_SETUP fails, should we use the non-short circuiting &=? #WontFix

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dunno this is copied from the current behavior in develop. I tried to not actually change the functional tests just consolidate them. But I may have messed up. You make a good point about if setup fails though to not run the test. I don't do that right in my test_p shim ... #WontFix

//
// Cortana Activation Tests
//
class CortanaVoiceCommandForegroundActivation {

@aballway aballway Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

takes me back 😿 #Closed

std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> GetTestMetaFactories() {
std::vector<std::pair<std::string, TestMetaFactoryBase<typename TestCase::ParamType>*>> toReturn;
for (typename TestInfoContainer::iterator test_it = tests_.begin(); test_it != tests_.end(); ++test_it) {
linked_ptr<TestInfo> test_info = *test_it;

@jaredhms jaredhms Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linked_ptr [](start = 6, length = 20)

nit: linked_ptr& #Resolved

// Pass null to indicate default app and app delegate classes
UIApplicationInitialize(nullptr, nullptr);
}

@jaredhms jaredhms Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shouldn't be resurrected; we now call UIApplicationInitialize in FunctionalTestHelpers::FunctionalTestSetupUIApplication() #Resolved

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah sorry bad merge on this file. I just wanted to delete everything in it basically but y'all made a bunch of changes to it. #Resolved


class ActivatedAppReceivesToastNotificationTests {
BEGIN_TEST_CLASS(ActivatedAppReceivesToastNotificationTests)
END_TEST_CLASS()

@jaredhms jaredhms Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: anywhere we're not adding class-level metadata, we can just use TEST_CLASS(ActivatedAppReceivesToastNotificationTests) (no need for begin/end) #WontFix

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know but I'm not changing how it was for people already. This is just transplanted code. #lazy. #WontFix

// 2. Use the same mechanism as in TEST_METHOD below to export a method from the WinObjC test file and call it here.
// 3. If you do not need this functionality feel free to remove this for your test class.
TEST_CLASS_CLEANUP(SampleTestClassCleanup) {
return true;

@jaredhms jaredhms Jan 27, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should return FunctionalTestCleanupUIApplication #Resolved

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk #Resolved

_EbrGetBaseWriteablePath

; REMOVE / CLEANUP C++ EXPORTS
?format@string@woc@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PBDZZ

@msft-Jeyaram msft-Jeyaram Jan 27, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format@string@woc@@ya?AV?$basic_string@DU?$char_traits@D@std@@v?$allocator@D@2@@std@@Pbdzz [](start = 9, length = 90)

I assume this mangled export is still needed? #WontFix

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably? Despite outward appearances, my goal is not to fix everything potentially wrong with this project. #WontFix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although somewhat disappointing.

@msft-Jeyaram

msft-Jeyaram commented Jan 27, 2017

Copy link
Copy Markdown

// Copyright (c) 2016 Microsoft Corporation. All rights reserved.

nit: update this #Resolved


Refers to: Frameworks/include/ErrorHandling.h:3 in 53f7de8. [](commit_id = 53f7de8, deletion_comment = False)

Comment thread tests/Tests.Shared/test-api.h Outdated
} \
catch (const ::WEX::TestExecution::VerifyFailureException&) \
{ \
} \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DHowett-MSFT @jaredms this means that I should bail out and log a failure here right? Test Cleanup should still run though? Basically put the TestBody in the first try block?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think so. gtest tries them all (?) in their own exception handlers i think.

auto factory = metaFactory.second->CreateTestFactory(*param); \
auto test = factory->CreateTest(); \
\
WEX::Logging::Log::StartGroup(currentTest.c_str()); \

@bbowman bbowman Jan 27, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WEX::Logging::Log::StartGroup(currentTest.c_str()); \ [](start = 20, length = 53)

would be useful to also log what the parameters used here are. I think gtest has a method for that somewhere. I dunno if I can get to it off of the ParamGenerator though?

size_t i = 0; \
static std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; \
static const std::wstring c_stringPrefix = L #prefix L"_" L#className L"." + converter.from_bytes(metaFactory.first.c_str()); \
for (typename ::testing::internal::ParamGenerator<className::ParamType>::iterator param = paramGenerator.begin(); param != paramGenerator.end(); ++param, ++i) { \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ypename ::testing::internal::ParamGeneratorclassName::ParamType::iterator [](start = 23, length = 76)

probably use auto& here?

Comment thread tests/Run-UnitTests.ps1
<#
.SYNOPSIS
Runs Tests for Islandwood project.
Runs Functional Tests for Islandwood project.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just merge this with run-functionaltests? Would be nice to only "maintain" 1 script.
Granted, once these are written, they usually don't have to be modified.

@pradipd

pradipd commented Feb 7, 2017

Copy link
Copy Markdown
Contributor

This is all sorted out.
You should be able to use the Run TAEF and Publish Task in VSTS.


In reply to: 275577758 [](ancestors = 275577758,275574435,275574352,275573399)

@bbowman bbowman force-pushed the bbowman/updateTAEF+RedoUTs branch from 7fe162c to 1a7732e Compare February 8, 2017 22:13
@bbowman

bbowman commented Feb 8, 2017

Copy link
Copy Markdown
Member Author

//******************************************************************************

Looks like the test_p benchmark tests fail... Is it possible that you aren't using standard GTEST macros for that?


Refers to: tests/Benchmark/EntryPoint.cpp:1 in a3246d1. [](commit_id = a3246d1, deletion_comment = False)

@@ -54,7 +48,22 @@ class CommandLineDrawingTestConfigImpl : public DrawingTestConfigImpl {

public:
CommandLineDrawingTestConfigImpl(int argc, char** argv)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the TestConfig base class is pure virtual so that we don't need to clowncar all of the TAEF behaviours into the "command line" one ;P

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like: there are so many ifdefs in this file it might simply be easier to have a win32 file that has its own config impl and the os x file that keeps the commandline one.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree in principle. I'll let you handle that. There were already a number of ifdefs in there for WIN32 so it seems like we need to think more about how to divide up the entrypoint code for the two.

std::string tempBuffer;
uint32_t maxPath = _MAX_PATH;
tempBuffer.resize(maxPath);
_NSGetExecutablePath(&tempBuffer[0], &maxPath);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces a dependency on Foundation, which not all of our test modules can take. We also cannot take that dependency on OS X.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only for OSX and it is what was happening before anyway somewhere ... I didn't make up that code at least.

@aballway

aballway commented Feb 8, 2017

Copy link
Copy Markdown
Contributor

The only macros it uses are TEST_P and INSTANTIATE_TEST_CASE_P, but it still has the GTEST current_test_info()->name(), which is probably causing the crash. I'm building it now so I'll see if there are any other things that would cause an issue. Thanks for fixing everything else up!

__if_not_exists(GetTestFullName) {
static std::string GetTestFullName() {
std::string toReturn;
return toReturn + ::testing::UnitTest::GetInstance()->current_test_info()->test_case_name() + "." +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add toReturn+ here? It looks like it's being done for std::string coercion? That's not great.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that is what it was for. I agree not great. I'll change.

@bbowman

bbowman commented Feb 8, 2017

Copy link
Copy Markdown
Member Author

I changed the test info name dealio and it seems to all pass for whatever that means...


In reply to: 278493789 [](ancestors = 278493789)

Comment thread .gitattributes
*.xcf filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.nupkg filter=lfs diff=lfs merge=lfs -text

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intended to be part of this change or part of the packaging change?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change does bring in the TAEF nupkg.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes.

Comment thread Frameworks/Starboard/EbrFile.cpp Outdated
const std::wstring& _IwGetWritableFolder() {
return g_WritableFolder;
unsigned int rawLength;
return WindowsGetStringRawBuffer(basePath.Get(), &rawLength);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

length is marked _Out_opt_ in the header; don't make it populate something you're discarding immediately.


// Set our writable and temp folders
IwSetWritableFolder(Windows::Storage::ApplicationData::Current->LocalFolder->Path->Data());
// Set our temp folders

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: not plural anymore

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the single line code, the comment is not necessary anymore (in the past, this used to be several lines :))


#include <tuple>

// Copying these macros from WexTestClass.h, but not including

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this comment, the following macros could simply be destroyed? Or does this exist for cross-platform support?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't look at that. I assume those run cross-plat so i think we should keep as is.

SB_EXPORT int EbrGetWantedOrientation();

SB_EXPORT const wchar_t* IwGetWritableFolder();
SB_EXPORT void IwSetWritableFolder(const wchar_t* folder);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd personally prefer to see all the Iw Writable Folder changes in a separate pull request with a separate commit (instead of squashed all together at the end). It will help future bisecting if we regress something.

In fact, I think all framework changes should be separate from this testing-related PR.


// TODO::
// todo-nithishm-04182016 - Investigate a better way to support ASSERT_*_MSG and EXPECT_*_MSG macros without re-defining them
// here. Bug 7244185 is tracking this work.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we port this VSO item to a GitHub item?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just remove the TODO. If we didn't need it after so long, we don't really need it :)

#define FRIEND_TEST(test_case_name, test_name) friend class test_case_name##_##test_name

// clang-format off
template <unsigned NumChar>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment explaining this and why we need it would be excellent.

str1[7] == 'D') ? L"TRUE" : L"FALSE";
}

__declspec(noinline) inline std::string _ExchangeAlternateTestName(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

??? noinline inline

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra special inlining. It means __declspec(selectany) for inline functions.

https://blogs.msdn.microsoft.com/freik/2005/10/26/how-to-make-a-no-inline-inline-function-and-why-you-might-want-to/ has a good summary.

The gist of it is that I'm using a global variable that needs to be shared across TUs (DrawingTest and the Test file that derives from it). This means I can't have more than one static function stamped out / use an inline function because the storage for the alternate name wouldn't be the same across TUs. However, I wanted to keep this in the header to minimize the additional goop needed by the tests (an alternate solution would be to place this stuff in the logger static lib). This means I need a "selectAny" type of behavior which you get from this hilarious syntax.

Comment thread tests/Tests.Shared/test-api.h Outdated
} \
catch (const ::WEX::TestExecution::VerifyFailureException&) \
{ \
} \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think so. gtest tries them all (?) in their own exception handlers i think.

{ \
static std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; \
std::wstring wideKey = converter.from_bytes(key); \
std::wstring wideValue = converter.from_bytes(value); \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Log::Property requiretypeof(value) == std::wstring? If not, and it's overloaded, we should pass value through unharmed for the appropriate implicit conversions to happen. This is too narrowly-specified as-is, it will only work for utf8 strings.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it requires wchar_t*s

const char* testName = _STRINGIFY(__##test_class); \
auto fullName = std::string(::testing::UnitTest::GetInstance()->current_test_info()->name()) + testName; \
::benchmark::internal::__RunCase(*m_test, fullName.c_str()); \
::benchmark::internal::__RunCase(*m_test, GetTestFullName().c_str()); \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetTestFullName() [](start = 46, length = 17)

I'll let you figure that out later. I don't think you do / you may want to rethink that.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 TestFullName is the value this needs, you're right

@DHowett-MSFT

Copy link
Copy Markdown

(please file issues for anything you're marking as "you can deal with this later"; otherwise, we will all collectively lose track of them until the one day where we're like "hey wait this bug seems familiar ...")

@bbowman

bbowman commented Feb 8, 2017

Copy link
Copy Markdown
Member Author

yep. Currently planning to file one on DrawingTests to integrate with the change (seems like we need to validate the arguments, perhaps adjust the ifdefs, and rework the test runner) and one for Benchmark tests


In reply to: 278498755 [](ancestors = 278498755)

Comment thread .gitattributes
*.xcf filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.nupkg filter=lfs diff=lfs merge=lfs -text

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change does bring in the TAEF nupkg.


// Set our writable and temp folders
IwSetWritableFolder(Windows::Storage::ApplicationData::Current->LocalFolder->Path->Data());
// Set our temp folders

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the single line code, the comment is not necessary anymore (in the past, this used to be several lines :))

using namespace Microsoft::WRL;
using namespace Windows::Foundation;

Wrappers::HString GetAppDataPath() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_IwGetWritablePath() won't work here?

@bbowman bbowman Feb 9, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will but I didn't really want to introduce that dependency to Starboard since its a weird private function anyway. I can though if we feel thats nicer? #Pending

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your choice, it was a nit from my side.

# Decide where the WTL output files will live
if ($WTLOutputDirectory -ne [string]$null)
{
$WTTLogFullPath = Join-Path $WTTLogPath wttlog.dll

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where would one get wttlog.dll?

@bbowman bbowman Feb 9, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from a valid location. We can take this offline. #Closed

Comment thread tests/Run-UnitTests.ps1 Outdated
ExecTest $argList

exit $failureCount + $crashingTestArray.length
exit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exit $script:exitCode

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://winobjc.visualstudio.com/WinObjC/_build/explorer?_a=summary&definitionType=2&buildId=7825

fails. It has some errors from CG.


In reply to: 100258200 [](ancestors = 100258200)

@pradipd

pradipd commented Feb 9, 2017

Copy link
Copy Markdown
Contributor

:shipit:

@DHowett-MSFT DHowett-MSFT left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd still like to see the non-testing changes moved out into another pull request. I want that so much that I'm even willing to do the work necessary to tease them apart. 😦

@bbowman bbowman force-pushed the bbowman/updateTAEF+RedoUTs branch from e2bbce9 to eba6f61 Compare February 9, 2017 22:04
@bbowman bbowman force-pushed the bbowman/updateTAEF+RedoUTs branch from eba6f61 to 9bdbefd Compare February 9, 2017 22:23
@bbowman bbowman force-pushed the bbowman/updateTAEF+RedoUTs branch from a5f35bf to 4940d88 Compare February 10, 2017 02:59
@bbowman bbowman merged commit 58e461e into microsoft:develop Feb 10, 2017
@DHowett-MSFT

Copy link
Copy Markdown

Woot

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants