Skip to content

fix: improve the performance of Meteor on Windows 10 and newer #13203

Merged
nachocodoner merged 8 commits intometeor:release-3.0.3from
ayewo:fix/windows-perf-improvement
Sep 10, 2024
Merged

fix: improve the performance of Meteor on Windows 10 and newer #13203
nachocodoner merged 8 commits intometeor:release-3.0.3from
ayewo:fix/windows-perf-improvement

Conversation

@ayewo
Copy link
Copy Markdown
Contributor

@ayewo ayewo commented Jun 24, 2024

This PR intends to close #12935.

It could probably be characterized as a 🕷 bug fix of sorts rather than a performance fix because even though Meteor on Windows is plagued mainly by two things:

  • slowdowns from Windows Defender and;
  • UNIX-style file access patterns which is non-idiomatic and slower on Windows;

my PR focuses more on mitigating the effects of Windows Defender.

Mitigating the effects of Meteor's UNIX-style design would require a significant refactoring of the code base.

Reproduction Repo

Based on my explanation above and following the guidance laid out in CONTRIBUTING, it made more sense to include my changes in a separate repo instead of trying to cram them here:

Summary

Overall, I believe this PR meets all of the Success Criteria:

  • 1. Root Cause Analysis:
    • Identify the specific reasons contributing to the slowness and intermittent failures of the meteor update on Windows.
    • Provide a detailed analysis of how these issues differ from the performance on Linux and MacOS.
  • 2. Performance Improvement:
    • Implement solutions that significantly improve the performance and stability of meteor update and package installations on Windows.
    • Ensure that these improvements do not negatively impact the performance of other operating systems (Linux and MacOS).
  • 3. Testing and Validation:
    • Demonstrate through testing that the improvements are effective and consistent across various Windows environments.
    • Include benchmarks or metrics showing the performance improvement compared to the previous version.
  • 4. Documentation and Reporting:
    • Provide comprehensive documentation of the changes made, including technical details and reasons for specific approaches.
    • Outline steps for reproducing the improvements and any necessary environment setups or configurations.
  • 5. Maintainability and Compatibility:
    • Ensure that the improvements are maintainable and do not introduce new bugs or vulnerabilities.
    • Verify compatibility with the latest version of Meteor.js (3.0) and widely-used packages.

/claim #12935

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Jun 24, 2024

CLA assistant check
All committers have signed the CLA.

@ayewo ayewo force-pushed the fix/windows-perf-improvement branch from 20ecf66 to 117dc84 Compare June 28, 2024 20:14
@ayewo ayewo requested a review from radekmie July 11, 2024 16:37
@netlify
Copy link
Copy Markdown

netlify bot commented Aug 20, 2024

Deploy Preview for v3-meteor-api-docs canceled.

Name Link
🔨 Latest commit c71be3c
🔍 Latest deploy log https://app.netlify.com/sites/v3-meteor-api-docs/deploys/66c7a2233caf1d000853e18b

@netlify
Copy link
Copy Markdown

netlify bot commented Aug 20, 2024

Deploy Preview for v3-migration-docs canceled.

Name Link
🔨 Latest commit c71be3c
🔍 Latest deploy log https://app.netlify.com/sites/v3-migration-docs/deploys/66c7a2230c41b90008515776

@denihs
Copy link
Copy Markdown
Contributor

denihs commented Aug 20, 2024

Hey @ayewo, the PR looks good, so I'm approving it. But before merging it, I'll ask @nachocodoner for help testing everything here.

Also, could you benchmark npx meteor instead of npm install -g meteor? I'm curious to see the difference there.

denihs
denihs previously approved these changes Aug 20, 2024
@ayewo
Copy link
Copy Markdown
Contributor Author

ayewo commented Aug 20, 2024

@denihs awesome, thank you!

Also, could you benchmark npx meteor instead of npm install -g meteor? I'm curious to see the difference there.

Okay. I have a lot on my plate right now so probably wont be able to find time to do that until next week at the earliest.

@nachocodoner
Copy link
Copy Markdown
Member

nachocodoner commented Aug 22, 2024

@ayewo: Your detailed report is impressive; thank you for inspiring us with such detailed insight into your process of improving Windows performance.

While this PR will significantly improve Meteor installation performance, the overall command execution time remains unchanged if understood correctly. I don't know if originally this issue wants to get more gains into other processes, like creating a meteor app. However, it's important to note what you mentioned in your report, that Windows Defender can slow down all Meteor processes, so disabling it could provide additional gains.

There may also be options within the Meteor tool itself to enhance performance further and affect meteor tool commands like meteor create and others. I haven't explored this fully, but using more modern and faster Node tools could improve the development experience across all environments, including Windows and UNIX.

Great work, we can include this fix in our upcoming release, version 3.0.3.

nachocodoner
nachocodoner previously approved these changes Aug 22, 2024
Copy link
Copy Markdown
Member

@nachocodoner nachocodoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve as a change to improve the Meteor installer speed on Windows.

image

This was my test on the npm installer. Just uninstalled Meteor, removed lines that require -g option and run Measure-Command { node install.js }. It went from 106 seconds down to 47 seconds with the changes on this PR. 🚀

@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Aug 22, 2024

Would it be possible to tag a special release for this to test on Windows rather than me just cloning the PR? I'd love to be able to meteor update --release 3.0.2-13203 or something to test adding packages, creating new meteor create , etc.

@nachocodoner
Copy link
Copy Markdown
Member

nachocodoner commented Aug 22, 2024

Would it be possible to tag a special release for this to test on Windows rather than me just cloning the PR? I'd love to be able to meteor update --release 3.0.2-13203 or something to test adding packages, creating new meteor create , etc.

@ayewo Correct me if I am wrong. But, will the rest of tools be affected by this change?

It doesn't as far as I understand. Maybe with the trick of the Windows Defender disabled, @wreiske can experience already gains with his current setup, but it won't expect anything else from this change as it only affects to npx meteor@latest installation, not meteor create and so on.

@ayewo
Copy link
Copy Markdown
Contributor Author

ayewo commented Aug 22, 2024

Would it be possible to tag a special release for this to test on Windows rather than me just cloning the PR? I'd love to be able to meteor update --release 3.0.2-13203 or something to test adding packages, creating new meteor create , etc.

@ayewo Correct me if I am wrong. But, will the rest of tools be affected by this change?

It doesn't as far as I understand. Maybe with the trick of the Windows Defender disabled, @wreiske can experience already gains with his current setup, but it won't expect anything else from this change as it only affects to npx meteor@latest installation, not meteor create and so on.

@nachocodoner You are absolutely correct: this change only affects installation. It will not impact the other meteor commands. I don't think disabling Windows Defender completely is an option for him which is why I explored adding folder and process exclusions to Windows Defender mitigate its interference.

@ayewo ayewo dismissed stale reviews from denihs and nachocodoner via c71be3c August 22, 2024 20:40
@ayewo
Copy link
Copy Markdown
Contributor Author

ayewo commented Aug 22, 2024

@nachocodoner BTW, thanks for the effusive praise 😊.

@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Aug 23, 2024

Would it be possible to tag a special release for this to test on Windows rather than me just cloning the PR? I'd love to be able to meteor update --release 3.0.2-13203 or something to test adding packages, creating new meteor create , etc.

@ayewo Correct me if I am wrong. But, will the rest of tools be affected by this change?
It doesn't as far as I understand. Maybe with the trick of the Windows Defender disabled, @wreiske can experience already gains with his current setup, but it won't expect anything else from this change as it only affects to npx meteor@latest installation, not meteor create and so on.

@nachocodoner You are absolutely correct: this change only affects installation. It will not impact the other meteor commands. I don't think disabling Windows Defender completely is an option for him which is why I explored adding folder and process exclusions to Windows Defender mitigate its interference.

For the sake of the bounty, I was really hoping for the create and update and add and other tools to have better performance. At the very least, some sort of automation to pop an administrator prompt up to "whitelist" the directories required to get better performance, or if administrator prompt is needed for proper symbolic links, that should be implemented.

The amount of detail in and time in @ayewo 's report alone IMO should be compensated, but I can't seem to /tip 250 @ayewo ... I also looked everywhere in the algora dashboard and there's no way to send a tip. The Meteor team might need to install the algora app on the meteor repository to enable the commands for adding bounties, tipping, etc.

If someone can help improve the rest of the tools timings, e.g. the general every day using of meteor for development on windows, I am willing to create a new bounty specifically for those pieces, or possibly keep this bounty open as it's not fully resolved and figure out how to compensate @ayewo for resolving the first issue (installation time).

To clarify, @ayewo , thank you for your effort in improving the performance of getting started (installing meteor), and your incredible detail on why the other commands are slow. I don't want to close the bounty as fully resolved because there are still issues with performance as detailed on the original bounty success criteria. I 100% appreciate and value your time and would like to compensate you for it. If you are able to provide another method for me to send a TIP since algora on the meteor repo doesn't seem to be working, I am happy to do that & keep this bounty open. If you complete the other tool fixes, I will make sure the additional $500 on top of the $250 ($750 total) is credited to you.

@algora-pbc
Copy link
Copy Markdown

Hi @wreiske. We just added the tip for you, it should now appear on your bounty board. Cheers!

@algora-pbc
Copy link
Copy Markdown

algora-pbc bot commented Aug 23, 2024

🎉🎈 @ayewo has been awarded $250! 🎈🎊

@ayewo
Copy link
Copy Markdown
Contributor Author

ayewo commented Aug 23, 2024

Hi @wreiske,

First off, thanks for the kind words!

For the sake of the bounty, I was really hoping for the create and update and add and other tools to have better performance. At the very least, some sort of automation to pop an administrator prompt up to "whitelist" the directories required to get better performance, or if administrator prompt is needed for proper symbolic links, that should be implemented.

I totally understand your frustration but as I explained in the report, those commands are at the heart of Meteor. Optimizing them is a significant amount of work that wouldn't quite fit within a $500 bounty, which is why I tried to quantify how much of the slow down was due to Windows Defender. It turned out to be quite significant which is why my solution dwells on adding process and folder exclusions to Windows Defender to try to keep the antivirus happy.

If you look at the benchmarks, the run times with Windows Defender disabled (i.e. the column titled Windows (Defender❌)) versus Windows Defender enabled but with process and folder exclusions in place (i.e. the column titled Windows (Defender✔️)) were slowly converging. But ... the last cell on the bottom right is conveniently empty.

Well, that's because I also met the exact issue you clearly articulated here. Since the issue is fairly reproducible and has in fact been reported before, I need to dig deeper to figure out the root cause so I can fix it once and for all.

Unfortunately, I have so much on my plate right now that I haven't been able to take a deeper look.

If you are able to provide another method for me to send a TIP since algora on the meteor repo doesn't seem to be working, I am happy to do that & keep this bounty open.

I appreciate the gesture, thanks 🙏.

With respect to the tip command not working, it tends to happen occasionally.
I've already looped in one of the Algora founders so he can take a look. He goes by @zcesur here on GitHub.

@nachocodoner nachocodoner changed the base branch from devel to release-3.0.3 September 10, 2024 12:41
@nachocodoner nachocodoner merged commit 1e1c191 into meteor:release-3.0.3 Sep 10, 2024
@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Jan 27, 2025

Hi @wreiske,

First off, thanks for the kind words!

For the sake of the bounty, I was really hoping for the create and update and add and other tools to have better performance. At the very least, some sort of automation to pop an administrator prompt up to "whitelist" the directories required to get better performance, or if administrator prompt is needed for proper symbolic links, that should be implemented.

I totally understand your frustration but as I explained in the report, those commands are at the heart of Meteor. Optimizing them is a significant amount of work that wouldn't quite fit within a $500 bounty, which is why I tried to quantify how much of the slow down was due to Windows Defender. It turned out to be quite significant which is why my solution dwells on adding process and folder exclusions to Windows Defender to try to keep the antivirus happy.

@ayewo , I agree - After looking at your in-depth report, the work is worth well over the $500 bounty to dive into the full performance improvements. I would like to pay out the full bounty to you, as my experience working with Meteor 3.1.1 on Windows has been greatly improved from your additions to the project.

I followed the documentation here, https://docs.algora.io/bounties/workflow#pay-out-a-claim but there doesn't seem to be a reward button.

image

Can someone help point me in the right direction to get this reward paid out?

Thank you again for all of your hard work on this bounty.

@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Jan 27, 2025

/reward @ayewo

@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Jan 27, 2025

@algora-pbc - can you help? Thank you!

@ayewo
Copy link
Copy Markdown
Contributor Author

ayewo commented Jan 27, 2025

@wreiske thanks!

Perhaps you could try the command: /approve ?

The full list of GitHub commands are here: https://docs.algora.io/commands

@wreiske
Copy link
Copy Markdown
Contributor

wreiske commented Jan 27, 2025

/approve

@algora-pbc
Copy link
Copy Markdown

Hi @wreiske - the reward button should appear once you log in with the email address you used to create the bounty

We'll improve this part of the flow very soon - thanks for your patience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG BOUNTY] $500 Reward to improve the performance of Meteor on Windows

8 participants