fix(assets): Use uint8arrays instead of Buffer in code that can run outside of Node by Princesseuh · Pull Request #9029 · withastro/astro

Princesseuh · 2023-11-08T22:08:24Z

Changes

Our service code can run outside of Node, so it shouldn't use Buffer, it should accept TypedArrays directly.

This is technically breaking, however:

It's extremely unlikely that someone relied on this being a Buffer when they made image services
As far as I know, no one has made actual full image services anyway
This corrects a flaw in the API, the way it was right now, you couldn't actually run things for real on non-Node hosts without making a service, a endpoint, spraying it with as any etc

It seems pretty safe to me, honestly. If we deem it not, I'm okay with delaying this to 4.0, no worries.

Fix withastro/adapters#54

Testing

Tests should still pass!

Docs

withastro/docs#5326

…utside of Node

changeset-bot · 2023-11-08T22:08:29Z

🦋 Changeset detected

Latest commit: c0b310f

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions

This PR is blocked because it contains a minor changeset. A reviewer will merge this at the next release if approved.

alexanderniebuhr · 2023-11-09T07:30:57Z

!preview assets-uint8array

github-actions · 2023-11-09T07:32:42Z

Snapshots have been released for the following packages:

astro@experimental--assets-uint8array
@astrojs/lit@experimental--assets-uint8array
@astrojs/markdown-remark@experimental--assets-uint8array
@astrojs/mdx@experimental--assets-uint8array

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--assets-uint8array tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/lit
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/node
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/prefetch
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vercel
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info astro is being published because our local version (0.0.0-assets-uint8array-20231109073138) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.0.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 3.0.0 is already published on npm
🦋  warn create-astro is not being published because version 4.5.0 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.3.1 is already published on npm
🦋  info @astrojs/lit is being published because our local version (0.0.0-assets-uint8array-20231109073138) has not been published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.7.1 is already published on npm
🦋  info @astrojs/mdx is being published because our local version (0.0.0-assets-uint8array-20231109073138) has not been published on npm
🦋  warn @astrojs/node is not being published because version 6.0.3 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.0.2 is already published on npm
🦋  warn @astrojs/preact is not being published because version 3.0.1 is already published on npm
🦋  warn @astrojs/prefetch is not being published because version 0.4.1 is already published on npm
🦋  warn @astrojs/react is not being published because version 3.0.4 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.0.3 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 3.0.2 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 4.0.3 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.0.2 is already published on npm
🦋  warn @astrojs/vercel is not being published because version 5.2.0 is already published on npm
🦋  warn @astrojs/vue is not being published because version 3.0.4 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.2.1 is already published on npm
🦋  info @astrojs/markdown-remark is being published because our local version (0.0.0-assets-uint8array-20231109073138) has not been published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.0.4 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.3.3 is already published on npm
🦋  info Publishing "astro" at "0.0.0-assets-uint8array-20231109073138"
🦋  info Publishing "@astrojs/lit" at "0.0.0-assets-uint8array-20231109073138"
🦋  info Publishing "@astrojs/mdx" at "0.0.0-assets-uint8array-20231109073138"
🦋  info Publishing "@astrojs/markdown-remark" at "0.0.0-assets-uint8array-20231109073138"
🦋  success packages published successfully:
🦋  astro@0.0.0-assets-uint8array-20231109073138
🦋  @astrojs/lit@0.0.0-assets-uint8array-20231109073138
🦋  @astrojs/mdx@0.0.0-assets-uint8array-20231109073138
🦋  @astrojs/markdown-remark@0.0.0-assets-uint8array-20231109073138
🦋  Creating git tags...
🦋  New tag:  astro@0.0.0-assets-uint8array-20231109073138
🦋  New tag:  @astrojs/lit@0.0.0-assets-uint8array-20231109073138
🦋  New tag:  @astrojs/mdx@0.0.0-assets-uint8array-20231109073138
🦋  New tag:  @astrojs/markdown-remark@0.0.0-assets-uint8array-20231109073138

Build Log


> root@0.0.0 build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/internal-helpers, @astrojs/lit, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prefetch, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/vercel, @astrojs/vue, @benchmark/timer, astro, create-astro
• Running build in 26 packages
• Remote caching enabled
::group::create-astro:build
cache hit, suppressing logs 895cd7a62cb990ee
::endgroup::
::group::@astrojs/internal-helpers:build
cache miss, executing 5189e4765f9441aa

> @astrojs/internal-helpers@0.2.1 build /home/runner/work/astro/astro/packages/internal-helpers
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::@astrojs/prism:build
cache miss, executing d2dcda1a046ff7cf

> @astrojs/prism@3.0.0 build /home/runner/work/astro/astro/packages/astro-prism
> astro-scripts build "src/**/*.ts" && tsc -p ./tsconfig.json

::endgroup::
::group::@astrojs/telemetry:build
cache miss, executing 470a5e5395d8d67a

> @astrojs/telemetry@3.0.4 build /home/runner/work/astro/astro/packages/telemetry
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdown-remark:build
cache miss, executing da9d79332a4a4061

> @astrojs/markdown-remark@0.0.0-assets-uint8array-20231109073138 build /home/runner/work/astro/astro/packages/markdown/remark
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::astro:build
cache miss, executing 293489ecd37ff434

> astro@0.0.0-assets-uint8array-20231109073138 build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" && tsc && pnpm run postbuild


> astro@0.0.0-assets-uint8array-20231109073138 prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"


> astro@0.0.0-assets-uint8array-20231109073138 postbuild /home/runner/work/astro/astro/packages/astro
> astro-scripts copy "src/**/*.astro" && astro-scripts copy "src/**/*.wasm"

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing 9a4c22834e330eca

> @astrojs/alpinejs@0.3.1 build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/lit:build
cache miss, executing e238051fad948f8f

> @astrojs/lit@0.0.0-assets-uint8array-20231109073138 build /home/runner/work/astro/astro/packages/integrations/lit
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing 7bed3c4b0c9f2502

> @benchmark/timer@0.0.0 build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing c59bfc9e6c5792e1

> @astrojs/tailwind@5.0.2 build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing 46dccb75be5e82a8

> @astrojs/react@3.0.4 build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing 919c5285edb89ede

> @astrojs/preact@3.0.1 build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing 034eb7ea3b7bcaf5

> @astrojs/solid-js@3.0.2 build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/node:build
cache miss, executing 2a62520071a46632

> @astrojs/node@6.0.3 build /home/runner/work/astro/astro/packages/integrations/node
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vercel:build
cache miss, executing f3648f8fcd2c43fc

> @astrojs/vercel@5.2.0 build /home/runner/work/astro/astro/packages/integrations/vercel
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing 9a704e8a904bb595

> @astrojs/markdoc@0.7.1 build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/prefetch:build
cache miss, executing 559c476c208b9a8a

> @astrojs/prefetch@0.4.1 build /home/runner/work/astro/astro/packages/integrations/prefetch
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing 1f81329a4c5dba6d

> @astrojs/partytown@2.0.2 build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing 5c7d1641861bdb54

> @astrojs/svelte@4.0.3 build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/rss:build
cache miss, executing 9fb0c309d3957904

> @astrojs/rss@3.0.0 build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing 21f3024ca82d0fc7

> @astrojs/underscore-redirects@0.3.3 build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing db65bd47ae565b8f

> @astrojs/mdx@0.0.0-assets-uint8array-20231109073138 build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing 2ff02b40539d4065

> @astrojs/vue@3.0.4 build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing 95d6c9f5d98c7a6b

> @astrojs/sitemap@3.0.3 build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    24 successful, 24 total
Cached:    1 cached, 24 total
  Time:    43.901s

alexanderniebuhr

I just tested this with a known good reproducible example and this fixes the issue, facing in non-node runtimes.

I also don't think this is breaking, so I would be very happy to get this out sooner than later.

/lgtm

ematipico

The changes look good to me. I believe this PR should be a patch because we used the wrong types (data structures) to handle binary data. For users affected by this wrong assumption from our end, we can provide a migration path to patch their code.

…r is important

Merge daaay

…utside of Node (#9029) * fix(assets): Use uint8arrays instead of Buffer in code that can run outside of Node * chore: changeset * docs: update changeset with more information on what to do if a Buffer is important * nit: do a patch instead

fix(assets): Use uint8arrays instead of Buffer in code that can run o…

5abaf98

…utside of Node

github-actions bot added the pkg: astro Related to the core `astro` package (scope) label Nov 8, 2023

chore: changeset

1026887

Princesseuh marked this pull request as ready for review November 8, 2023 22:24

github-actions bot added pr: docs A PR that includes documentation for review semver: minor Change triggers a `minor` release labels Nov 8, 2023

github-actions bot previously requested changes Nov 8, 2023

View reviewed changes

alexanderniebuhr approved these changes Nov 9, 2023

View reviewed changes

ematipico approved these changes Nov 9, 2023

View reviewed changes

Princesseuh mentioned this pull request Nov 9, 2023

[i18nIgnore] feat(image-service-reference): Change types to Uint8Array withastro/docs#5326

Merged

Princesseuh added 2 commits November 9, 2023 14:04

docs: update changeset with more information on what to do if a Buffe…

d21544b

…r is important

nit: do a patch instead

c0b310f

Princesseuh merged commit 29b83e9 into main Nov 9, 2023

Princesseuh deleted the fix/uint8array-assets branch November 9, 2023 13:05

astrobot-houston mentioned this pull request Nov 9, 2023

[ci] release #9021

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(assets): Use uint8arrays instead of Buffer in code that can run outside of Node#9029

fix(assets): Use uint8arrays instead of Buffer in code that can run outside of Node#9029
Princesseuh merged 4 commits intomainfrom
fix/uint8array-assets

Princesseuh commented Nov 8, 2023 •

edited

Loading

Uh oh!

changeset-bot bot commented Nov 8, 2023 •

edited

Loading

Uh oh!

github-actions bot left a comment

Uh oh!

alexanderniebuhr commented Nov 9, 2023

Uh oh!

github-actions bot commented Nov 9, 2023

Uh oh!

alexanderniebuhr left a comment

Uh oh!

ematipico left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Princesseuh commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Testing

Docs

Uh oh!

changeset-bot bot commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

alexanderniebuhr commented Nov 9, 2023

Uh oh!

github-actions bot commented Nov 9, 2023

Uh oh!

alexanderniebuhr left a comment

Choose a reason for hiding this comment

Uh oh!

ematipico left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Princesseuh commented Nov 8, 2023 •

edited

Loading

changeset-bot bot commented Nov 8, 2023 •

edited

Loading