Skip to content

Improve base64 encoding/decoding speeds#1985

Merged
FrederikBolding merged 7 commits intomainfrom
fb/fast-base64
Dec 4, 2023
Merged

Improve base64 encoding/decoding speeds#1985
FrederikBolding merged 7 commits intomainfrom
fb/fast-base64

Conversation

@FrederikBolding
Copy link
Copy Markdown
Member

@FrederikBolding FrederikBolding commented Nov 23, 2023

Adds new base64 encoding and decoding utilities that have faster performance characteristics than the current implementation. This also fixes an issue where the client would sometimes get unresponsive while encoding large files.

Fixes #1984

async base64 encoding of 98095974 bytes took 596 ms
sync base64 encoding of 98095974 bytes took 86004 ms
async base64 decoding of 98095974 bytes took 5525 ms
sync base64 decoding of 98095974 bytes took 73044 ms

@codecov
Copy link
Copy Markdown

codecov bot commented Nov 23, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (0bbe326) 96.08% compared to head (d05a3ed) 96.08%.

❗ Current head d05a3ed differs from pull request most recent head 5be4c26. Consider uploading reports for the commit 5be4c26 to get more accurate results

Files Patch % Lines
packages/snaps-utils/src/base64.ts 92.85% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1985      +/-   ##
==========================================
- Coverage   96.08%   96.08%   -0.01%     
==========================================
  Files         267      269       +2     
  Lines        6240     6250      +10     
  Branches     1009     1008       -1     
==========================================
+ Hits         5996     6005       +9     
- Misses        244      245       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines +2380 to +2386
await Promise.all(
auxiliaryFiles.map(async (file) => {
// This should still be safe
// eslint-disable-next-line require-atomic-updates
file.data.base64 = await asyncEncode(file);
}),
);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we doing this here? Wouldn't it mean we're storing files in memory multiple times?

Copy link
Copy Markdown
Member Author

@FrederikBolding FrederikBolding Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason for this was to move it to a function that was already asynchronous, therefore "precomputing" the base64 string before trying to set it in state. Alternatively we could turn #set into an async function and do it there, but I'm not sure its worth the extra complexity? WDYT?

I think the increase in time the file is stored in memory should be neglible?

@FrederikBolding FrederikBolding marked this pull request as ready for review December 1, 2023 10:31
@FrederikBolding FrederikBolding requested a review from a team as a code owner December 1, 2023 10:31
Mrtenz
Mrtenz previously approved these changes Dec 4, 2023
@FrederikBolding FrederikBolding merged commit e301bda into main Dec 4, 2023
@FrederikBolding FrederikBolding deleted the fb/fast-base64 branch December 4, 2023 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate slowness when installing with large auxiliary files

2 participants