154

I use Powershell's Invoke-WebRequest method to download a file from Amazon S3 to my Windows EC2 instance.

If I download the file using Chrome, I am able to download a 200 MB file in 5 seconds. The same download in PowerShell using Invoke-WebRequest takes up to 5 minutes.

Why is using Invoke-WebRequest slower and is there a way to download at full speed in a PowerShell script?

Improve this question

8 Answers 8

Reset to default
290

Without switching away from Invoke-WebRequest, turning off the progress bar did it for me. I found the answer from this thread: https://github.com/PowerShell/PowerShell/issues/2138 (jasongin commented on Oct 3, 2016)

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest <params>

For my 5MB file on localhost, the download time went from 30s to 250ms.

Note that to get the progress bar back in the active shell, you need to call $ProgressPreference = 'Continue'.

Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

For a 100MB file this reduces the time from 10 Minutes to 2 Seconds. I wish developers would think more about the architecture of their software.
A lesson on how not to implement a progress bar.
It appear the progress bar updates after every byte, which is utter madness.
Can't believe that -ProgressPreference isn't a parameter with Invoke-WebRequest
Windows hasn't changed one bit.
|
80

I was using

Invoke-WebRequest $video_url -OutFile $local_video_url

I changed the above to 

$wc = New-Object net.webclient
$wc.Downloadfile($video_url, $local_video_url)

This restored the download speed to what I was seeing in my browsers.

Improve this answer

5 Comments

So I just ran wc.downloadFile(ibm-s3-url, "./test.tar.gz"), it did something, presumably download that file, but it didnt put it in my working directory... any idea where it might have gone?
@Groostav Mine showed up in c:\windows\system32. Not exactly the first place I'd look when downloading to a relative location.
@Groostav the current process' working directory is not the same thing as PowerShell's current location.
Can you show progress with this approach?
I prefer this one: It lets me create the [System.Net.WebClient]::new() once, then run several downloads without recreating a new object (like Invoke-Webrequest does) which saves time. And I can set .AllowReadStreamBuffering = $true and AllowWriteStreamBuffering = $true. Just don't forget .Dispose() when done, at least if your script is a long runner you should cleanup created objects.
33

$ProgressPreference = 'SilentlyContinue' I got this down from 52min down to 14sec, for a file of 450 M. Spectacular.

Improve this answer

3 Comments

please include more information in your answer, eg where does the setting you describe get set/defined?
This is a built-in preference variable to PowerShell, and controls whether or not a progress bar is displayed for certain operations, such as downloading files via Invoke-WebRequest. The reason this improves performance here is because Invoke-WebRequest (and Invoke-RestMethod) count the downloaded bytes too often (every single byte I believe), so the cmdlet is actually slowed down as it tallies how many bytes have been processed.
Is this answer actually a "thank you" comment to this answer?
23

One-liner to download a file to the temp directory:

(New-Object Net.WebClient).DownloadFile("https://www.google.com", "$env:temp\index.html")
Improve this answer

6 Comments

How does this answer the question?
Lloyd asked: "is there a way to download at full speed in a PowerShell script?" This answer is a one-liner way to do that. Downloading to the temp directory is the canonical way to demonstrate this.
This is actually a good answer, WebClient.DownloadFile is much faster than Invoke-WebRequest due to the way Invoke-WebRequest tracks its own progress
Absolutely relevant, this fixes exactly the problem OP has (and mine :) )
I actually prefer this one, because you don't have to worry about clobbering $ProgressPreference or saving/restoring it. As long as you're simply downloading a file, I prefer the one-liner.
|
3

Unfortunately the progress bar of Invoke-WebRequest is slowing file download a lot on Windows Powershell 5.1 (the version included in Windows OS). It's much faster on later Powershell versions (I tested it on Powershell 7.3).

IMO, if you are forced to use Windows Powershell then the best way is to use curl since it's included on Windows by default now. Just be aware that by default Windows Powershell has alias named curl for Invoke-WebRequest so to run curl program you need to write curl.exe to tell Windows Powershell that you don't want to use curl alias.

This command takes 11 minutes on Windows Powershell 5.1 and 23 seconds on Powershell 7.3:

Invoke-WebRequest -Verbose -Uri "https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe" -OutFile ".\dotnet-sdk-6.0.410-win-x64.exe"

and this takes 15 seconds:

curl.exe -fSLo .\dotnet-sdk-6.0.410-win-x64.exe https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe
Improve this answer

2 Comments

Could you explain the -fSLo bit? Man pages say -f makes the command fail silently. That sounds undesirable. And I can't even find the -L flag.
@gargoylebident Check explainshell. It should explain params well. To be honest I don't remember now where I've found this particular set of options. I think it was in some script I saw on Github (for example like here).
2

I just hit this issue today, if you change the ContentType argument to application/octet-stream it is much faster (as fast as using webclient). The reason is because the Invoke-Request command will not try and parse the response as JSON or XML.

Invoke-RestMethod -ContentType "application/octet-stream" -Uri $video_url  -OutFile $local_video_url
Improve this answer

2 Comments

-ContentType is for POST requests only AFAIK so it probably won't make a difference.
You'd be better off using the -UseBasicParsing parameter but that won't make much of a difference for static files. Setting $ProgressPreference = 'SilentlyContinue' before calling Invoke-WebRequest greatly improves the performance.
-1

I had the same issue, I replace Invoke-WebRequest by CURL.exe call using Invoke-Expression and a command line in my script

went from 10 min to 40 sec of download

Improve this answer

1 Comment

This does not answer why one method is faster than the other
-1

$ProgressPreference = 'SilentlyContinue' I got this down from 30min to ~40 seconds.

Unbelievable, Microsoft, unbelievable...

Improve this answer

1 Comment

That's what two existing answers say already.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.