BTW: I have filed a bug with Microsoft regarding this issue at https://connect.microsoft.com/PowerShell/feedback/details/927384/progress-stream-being-sent-to-stderr-in-winrm-responses

I have two points here:

1. What implies set it to SilentlyContinue?
2. I made some changes to the WinRM branch and I see old powershell commands. Could you re-run bundle install and try again?

Thanks for posting this issue/PR...

I just synced up and this reproduces on every first kitchen verify after creating a box. Subsequent verifies succeed. Just curious, after a kitchen destroy, does your first kitchen verify fail? BTW: I'm using serverspec tests.

Happy to answer any questions on this.

Here is the relevant excerpt from help about_preference_variables

$ProgressPreference
-------------------
Determines how Windows PowerShell responds to progress updates
        generated by a script, cmdlet or provider, such as the progress bars
        generated by the Write-Progress cmdlet. The Write-Progress cmdlet
        creates progress bars that depict the status of a command.

        Valid values:
    Stop:               Does not display the progress bar. Instead,
                                it displays an error message and stops executing.

         Inquire:            Does not display the progress bar. Prompts
                                for permission to continue. If you reply
                                with Y or A, it displays the progress bar.

          Continue:           Displays the progress bar and continues with
          (Default)           execution.

          SilentlyContinue:   Executes the command, but does not display
                              the progress bar.

So SilentlyContinue will cause all writes to the progress stream (write-progress) to go to the unix equivalent of dev/null and thus winrm will not include that output in its responses.

Interesting.. I do not have that problem at all.. Let's try to isolate the issue!

Cloud you follow this simple Gist and create a new one with the output and share it with me?

• https://gist.github.com/afiune/aefda6e085eb8e524039#file-tk-windows-cookbook-test

This test cookbook is using kitchen-vagrant and serverspec as well..

Looks like that gist will not run on windows which is where I have been hosting my testing. I'm running it on my linux (ubuntu 14) box now but it hangs on installing ridley. I'm gonna kill it and try again but I'm not very confident. Do you have anything that can run on windows?

ok. after fiddling, i've got this running both on linux and windows. I'm running it with my Hyper-V vagrant box on windows since I cannot install VBox along side of it.

I get a new error early in the convergence:

>>>>>> Converge failed on instance <default-windows-2012R2>.
>>>>>> Please see .kitchen/logs/default-windows-2012R2.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: WinRM exited (1) using shell [powershell] for command: [sh -c '
chef-client --local-mode --config /tmp/kitchen/client.rb --log_level info --chef-zero-port 8889 --json-attributes /tmp/kitchen/dna.json
']
ERROR:
>>>>>> ----------------------

Looks like it is trying to run linux commands on the host.

On linux, I had to fanagle it to use my installed version of vagrant because it wanted to use it via a gem which vagrant did not like at all. Using your vbox image it did work.

I'm gonna dig into this and see what needs to be done to get it working on a windows host.

Thanks!

The gist has been tested on Windows/Unix host.. So it should work. About that error, I'm thinking what is happening because that shouldn't happen now! that's an old issue.. uhmm could you try to delete the Gemfile.lock and then run bundle install again? Thanks

The bundle install never fully worked on windows and I have had lots of problems with bundler on windows. It dies when it tries to compile native extensions. So I just copied the new fems over the old ones which I realize is a bit sketchy.

Here is the error from bundle install:

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    C:/opscode/chefdk/embedded/bin/ruby.exe extconf.rb
creating Makefile
/usr/lib/ruby/gems/2.0.0/gems/libyajl2-1.0.1/ext/libyajl2

make "DESTDIR="
make: Nothing to be done for `all'.

make "DESTDIR=" install
mkdir -p C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/libyajl2-1.0.1/lib/libyajl2/vendored-libyajl2/lib
      0 [main] mkdir 1604 open_stackdumpfile: Dumping stack trace to mkdir.exe.stackdump
make: *** [install] Error 5


Gem files will remain installed in C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/libyajl2-1.0.1 for inspection.
Results logged to C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/libyajl2-1.0.1/ext/libyajl2/gem_make.out
An error occurred while installing libyajl2 (1.0.1), and Bundler cannot
continue.
Make sure that `gem install libyajl2 -v '1.0.1'` succeeds before bundling.
C:\dev\tk-windows [master +1 ~1 -0 !]> cd .\dev\tk-windows

I'll troubleshoot this later, but I also wanted to let you know that today I deployed your test-kitchen winrm fork on my ubuntu chef workstation at work. There I have a custom vsphere chef-metal driver that I extended so that it can also serve as a kitchen driver. Things mostly worked but I had these issues:

1. Lots of intermittent failures from the progress stream writing to stderr on winrm (the issue this PR is intended to address.) I hit this on almost exactly every other call to kitchen verify.
2. Missing proper windows busser defaults in the configuration. I had to add this to my config:

busser:
  busser_bin: /tmp/busser/gems/bin/busser.bat
  sudo: false

These should probably be the default for windows guests. I did not hit this before you pulled in Fletcher Nicols commits so I'm guessing this is related to that.

Its interesting that I did not have issues with the progress stream on vbox. I did not really look deeply into it. Could be that vbox fundamentally does networking differently than hyper-v/vmware but I'm not sure why that would impact this issue.

I'd be happy to test/troubleshoot/send PR's as I bump into stuff. Testability on Windows is key to our organization and some personal projects of my own.

Great stuff @mwrock !!! I really like that you are testing this in different environments. For the first issue, I don't have a windows host to test but there is many people that is using it so I will pass this error to them to get some help.

The second one, (vmware over linux) I can confirm that the properties that you set are the default one 👍 So I am not sure why you had to set it manually. I agree that I had to refactor some stuff after that bid giant commit but I'm sure I solved everything.. So I might assume (as you don't use bundler) that the code is old and you need to pull it down.. (maybe)

Yeah, I'd be happy too. Please do it, continue sending them and I will track them down as we continue developing..

btw: I'm testing your PR on my MAC! 👍

Could you validate that you have this commit [https://github.com/afiune/test-kitchen/commit/9a7545b613e655ce65a642e99b5e3cf2c2b92428] applied on you local computer where you are testing..

Hey @btm could you help me with this? I know you tested my code on Windows7/Windows8 with ChefDK and without it..

Thanks in advance!

Hey @afiune, on my ubuntu box, I did a gem build and then a gem install so that the change would be globally available because I enjoy living dangerously. After doing so a Kitchen -v shows I am using 1.3.0. while the previous version installed via chefdk is 1.2.1/1.2.2.dev.

I admit that I am still far from comfortable with the nuances of ruby versioning conventions enforced by gems and bundler but would it be possible that that would not be sufficient? I did confirm that 1.3.0 includes the commit referenced above.

@mwrock in your bundle install error, I see:

0 [main] mkdir 1604 open_stackdumpfile: Dumping stack trace to mkdir.exe.stackdump

I encountered similar errors with tar.exe when I was working on getting gecode/berks 3 running on windows. In my case, the problem was caused by a faulty copy of tar.exe that was bundled with the windows git package. For reference: berkshelf/berkshelf#1184 (comment)

In any case, I wonder where this copy of mkdir.exe that is creating stack dumps comes from. Can you track it down? Use which mkdir or where mkdir to find the executable.

HTH.

@danielsdeleo I think the issue is that it is calling

mkdir -p C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/libyajl2-1.0.1/lib/libyajl2/vendored-libyajl2/lib

and the windows mkdir command does not have a -p parameter.

just synced up my branch cleaning up the conflicts.

ok i found the missing link on my windows box. I run chef via chefdk. chefdk calls the test-kitchen bits from embedded/apps and that was not updated. After copying the latest test-kitchen/lib over apps/test-kitchen/lib, things are working again. However, if I remove my edit to winrm_base, I continue to get errors like this:

-----> Running postinstall for serverspec plugin
>>>>>> Setup failed on instance <default-windows-2012R2>.
>>>>>> Please see .kitchen/logs/default-windows-2012R2.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: WinRM exited (0) using shell [powershell] for
            command: [$env:BUSSER_ROOT="/tmp/busser"; $env:GEM_HOME="/tmp/busser/gems"; $env:GEM_PATH="/tmp/busser/gems"; $env:PATH="$env:PATH;$env:GEM_PATH/bin"; try { $env:BUSSER_SUITE_PATH=@(busser suite path) } catch { $env:BUSSER_SUITE_PATH="" }; $env:GEM_CACHE="/tmp/busser/gems/cache"; If ((gem list busser -i) -eq "false") {
          gem install busser --no-rdoc --no-ri }; busser plugin install busser-serverspec]
REMOTE ERROR:

>>>>>> ----------------------
C:\dev\Boxstarter-cookbook\boxstarter [master]>

Note the 0 error code and no error message. The raw winrm response is:

{:data=>[{:stderr=>"#< CLIXML\r\n", "<Objs Version=\"1.1.0.1\" xmlns=\"http://schemas.microsoft.com/powershell/2004/04\"><Obj S=\"progress\" RefId=\"0\"><TN RefId=\"0\"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N=\"SourceId\">1</I64><PR N=\"Record\"><AV>Preparing m", "odules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S=\"progress\" RefId=\"1\"><TNRef RefId=\"0\" /><MS><I64 N=\"SourceId\">2</I64><PR N=\"Record\"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>"}], :exitcode=>0}

You can easily reproduce this in irb:

require 'winrm'
require 'kconv'

conn=WinRM::WinRMWebService.new('http://172.172.172.172:5985/wsman', :plaintext, :user => 'administrator', :pass => 'Pass@word1', :basic_auth_only => true)
conn.powershell("write-progress 'hi'")

Unless you have a winrm client customized to parse out these progress messages, they are invisible to a winrm session. These are those progress bar messages in light blue that you sometimes see in a powershell console. Since the content in the context of a test-kitchen run is immaterial to the actual test run, the easiest way to get around this is to simply turn off the progress stream output.

Hey you wanna test the new Transport Abstraction? It's Released!!

Transport Branch!!

https://github.com/afiune/tk-multiplatform
https://github.com/afiune/tk-windows
https://gist.github.com/afiune/486430a0e60e0544563f

Oh I am SO on it!

So this is slightly better and perhaps a good move as a "default" implementation. Using my "wrapper" cookbook with direct dependencies on:
windows
newrelic
ms_dotnet4

Note: The above list does not include transitive dependencies. There are quite a few.

Setting max_threads to 20 resulted in a download time of 6 minutes. Unfortunagely this was only a couple minutes faster than 1 thread.

Adjusting max_threads to 28 did not make a difference. Any higher and the default shell count limit is exceeded (the limit is 30 but I think a shell or two are used on the side) and its not a good idea to change that.

Obviously 6 minutes is still way to long to wait. What I like about the new transport model is that I can now create my own vsphere_winrm transport class that leverages the vsphere API and can bring the time down to a couple seconds.

I'd think that you could do something with synced folders to get the vagrant plugin down to a similar number. Adding a synced folder on the sandbox directory parent and then just do a Copy-Item in powershell via winrm.

One thing to include in error handling would be to make sure connections are closed. I had to manually kill all the winrs processes on the windows node. Also (and this is an issue with the winrm gem) the "real" error when the limit is exceeded is hidden and the user only gets a generic 500 message with nothing actionable. I had to hack the winrm gem to see the true error (I could have done the same by sniffing the http traffic). I have filed an issue WinRb/WinRM#83 to address that.

Before I refactor my fork and create a new transport, can you tell me if the basic structure here is pretty stable or would you suggest holding off to let the dust settle?

Thanks!!

Hey @mwrock pretty good catch!! Thanks for opening that ticket ... Let's see if @sneal has some comments about it!

I know this new transport abstraction is going to be an awesome way to implement your communication methods, just like you do with drivers and provisioners, that's the whole idea!

Talking about the upload process I think it should stay like that since test-kitchen has many other drivers and not only kitchen-vagrant. I think @fnichol can support that thought.. ✌️

I would say hold on just next week to get the approval from @fnichol... But you can start doing some work around it since I really don't think it will change to much. 👍

Cool. I had another thought on the "default" winrm implementation: With powershell remoting you can transfer files like this: http://www.powershellcookbook.com/recipe/ISfp/program-transfer-a-file-to-a-remote-computer This essentially streams the binary content and I don't think it does so by sending several requests, but maybe it does. I've used it with some large transfers and it is much faster than what we are seeing here.

Powershell remoting is built on top of winrm and I'm not certain off the top of my head how this could be implemented in straight up winrm but it would be interesting to reverse engineer the http messages and come up with a winrm implementation that does the same. It may mean needing to extend the winrm gem but that would be a cool feature if its feasible.

I have not looked closely myself yet and it is possible my payloads were much smaller than the set of cookbooks I am dealing with here and it actually is sending several requests. Even so, you could still achieve large optimizations by compressing to a single file and using much larger chunks than the 8k here.

Might be worth a look...

Nice! I will take a look tomorrow ...

Just took a look at a 45mb file copy in wire shark. It does make several posts but they are around ~180k each.

I briefly looked closer at this today and it looks quite doable to get some significant optimizations here. First, the easiest way to reverse engineer this is from a Windows client.

• Ajust your local winrm settings to allow unencrypted traffic:

Set-Item wsman:\localhost\client\AllowUnencrypted -Value $true

• Using the script I linked to above, when you create the remote session, adjust session options to turn off encryption and compresion traffic.

$opts = New-PSSessionOption -NoCompression -NoEncryption
$session = New-PSSession myguest -Credential $creds -SessionOption $opts

So looking at this I noticed that the winrm stream had a command element and an argumants element. You can see here the small script included in the link above but the bytes that get piped into that script are in the .

<s:Body><rsp:CommandLine xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" CommandId="73A475CA-6F7C-49E1-8A99-629B0532030E"><rsp:Command>
    param($destination, $bytes)
    ## Convert the destination path to a full filesystem path (to support
    ## relative paths)
    $Destination = $executionContext.SessionState.`
        Path.GetUnresolvedProviderPathFromPSPath($Destination)
</rsp:Command><rsp:Arguments>AAAAAAAAAJ8AAAAAAAAAAAMAABcAAgAAAAYQAgDUC3jtBv7hQ4ORDecnYkrqynWkc3xv4UmKmWKbBTIDDu+7vzxPYmogUmVmSWQ9IjAiPjxNUz48T2JqIE49IlBvd2VyU2hlbGwiIFJlZklkPSIxIj48TVM+PE9iaiBOPSJDbWRzIiBSZWZJZD0iMiI+PFROIFJlZklkPSIwIj48VD5TeXN0ZW0uQ29sbGVjdGlvbnMuR2VuZXJpYy5MaXN0YDFbW1N5c3RlbS5NYW5hZ2VtZW50LkF1dG9tYXRpb24uUFNPYmplY3QsIFN5c3RlbS5NYW5hZ2VtZW50LkF1dG9tYXRpb24sIFZlcnNpb249My4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj0zMWJmMzg1NmFkMzY0ZTM1XV08L1Q+PF
...

I'm betting that the arguments are not bound to a size limit.

What is also interesting here is that the powershell script is not wrapped in a cmd.exe call to powershell so the arguments are actual powershell arguments. I'm guessing this is done because the winrm endpoint is http://192.168.1.14:5985/wsman?PSVersion=4.0. Note the PSVersion query param. A cursory search did not yield any further info on this param.

So the base winrm gem would need to be adjusted to accomodate a "pure" ps shell but even without that you could take advantage of calling the winrm run_command which does have an arguments parameter. I'd imagine you would simply pass powershell as the command and pass the EncryptedCommand as an argument. So you could compress to a single file and chunk it into ~1MB chunks that you would append to in each call and then uncompress as a final step.

Not sure what your priorities look like, but I'd be happy to take a stab at this this week if you are in the middle of other stuff. Otherwise just keep me posted on how things are progressing.

This is all very exciting stuff to see convergence testing becoming a reality on Windows and thanks a bunch for the good work here!!

I'd like to see the winrm gem contain some sort of remote file system abstraction. It would reduce duplication between this and Vagrant. @mwrock Interesting findings about larger file sizes. I'm going to look at that.

+1 on handling this from the winrm gem. Some kind of file_copy method exposed from their would be great!