Skip to content

Conversation

@Richienb
Copy link
Member

@Richienb Richienb commented Nov 10, 2019

This PR replaces the encoding optional dependency with fetch-charset-detection.

Fixes #554.

cc @bitinn @xxczaki

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
@Richienb Richienb requested review from bitinn and xxczaki November 10, 2019 04:19
Copy link
Collaborator

@bitinn bitinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's my fault to not state this explicitly:

But my original intention was for us to drop encoding and charset detection from node-fetch completely.

As in, we won't depends on fetch-charset-detection neither, people who want this feature will need to add this dependency to their code.

That's what I meant by deprecation. Also what I meant by breaking changes.

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

Also the diff view on GitHub is weird, I wouldn't expect large change on package.json, also writeToStream is somehow removed?

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

Also note the charset detection is not a part of the Fetch Spec, which is why we have been trying to deprecate it. I introduced them in v1 and I regret it ever since.

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

There is no reason is hook charset detection into our code neither, by doing encoding conversion you essentially need to cache the whole response in memory. Get them through res.body is basically the same.

Also this is why I asked about documentation, we are introducing a breaking API changes, people who use node-fetch as web crawler will need to use the new package, so documentation is needed to guide them on the upgrade.

(I am writing these comments for future reader's reference, because obviously people will hate breaking changes)

@Richienb
Copy link
Member Author

Also this is why I asked about documentation, we are introducing a breaking API changes, people who use node-fetch as web crawler will need to use the new package, so documentation is needed to guide them on the upgrade.

Charset encoding detection is already described in the v3 migration guide and the API is documented in the documentation.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
@Richienb
Copy link
Member Author

@bitinn Just for clarification, which functions were we supposed to migrate to fetch-charset-detection?

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

Public API-wise it's really just textConverted, I do not believe we expose anything else. But we should remove code that's added only in support for textConverted, like convertBody.

I am not aware of any other dependency on encoding.

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

Given v2 code I would say fetch-charset-detection could expect a buffer input.

https://github.com/bitinn/node-fetch/blob/master/src/body.js#L151

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

Now that I have looked at fetch-charset-detection, I think it's doing a lot more than my original expectation. Sorry for not clarifying early. I would have keep fetch-charset-detection minimal (and it's should be because it doesn't depends on anything else, just a buffer input, and a content-type header)

@bitinn
Copy link
Collaborator

bitinn commented Nov 13, 2019

In short, I would just ask the users to pass the res.buffer and content-type header. fetch-charset-detection need not to know node-fetch, to be honest. It can be used for any other response that's compatible.

@Richienb Richienb changed the base branch from 3.x to master November 17, 2019 02:12
@Richienb Richienb changed the base branch from master to 3.x November 17, 2019 02:12
…r others.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
…nn/node-fetch into add-fetch-charset-detection

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
Signed-off-by: Richie Bendall <richiebendall@gmail.com>
@Richienb Richienb requested a review from bitinn November 17, 2019 04:57
@Richienb
Copy link
Member Author

@bitinn It's ready for another re-review.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
Copy link
Collaborator

@bitinn bitinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I hope there is now no more confusion.

package.json Outdated
},
"optionalDependencies": {
"encoding": "^0.1.12"
"fetch-charset-detection": "^0.2.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if I can be more clear, but no, we don't want dependency on fetch-charset-detection at all, not even optional

src/body.js Outdated
let convertBody;
try {
convert = require('encoding').convert;
convertBody = require('fetch-charset-detection').convertBody;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which means, we are removing these code, not replacing them with the new code

src/body.js Outdated
* @return Promise
*/
textConverted() {
if (!convertBody) {
Copy link
Collaborator

@bitinn bitinn Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which means, we no longer has this API, we dropped it.

the users will have to rewrite their code from res.textConverted() to something like

var convertBody = require('fetch-charset-detection');

var buffer = res.buffer();
var type = res.headers.get('content-type');
var text = convertBody(buffer, type);

@Richienb
Copy link
Member Author

Oh. I am very stupid. I'll be back for another review.

@bitinn
Copy link
Collaborator

bitinn commented Nov 17, 2019

@Richienb it's my bad, I should have been more clear on this. I thought it should be clear while the situation is more nuanced than my expectation.

@Richienb Richienb requested a review from bitinn November 17, 2019 09:14
Copy link
Collaborator

@bitinn bitinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit of nitpicking, it's otherwise fine.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>
@Richienb Richienb requested a review from bitinn November 18, 2019 03:14
Copy link
Collaborator

@bitinn bitinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now

@bitinn
Copy link
Collaborator

bitinn commented Nov 18, 2019

BUT: our tests keep failing, can we fix that first? Is it pending on some PR?

@xxczaki xxczaki mentioned this pull request Nov 18, 2019
35 tasks
@Richienb
Copy link
Member Author

@bitinn The failing build is related to Replace the calls to url.parse and url.resolve with calls to new URL to use the new UTF-8 awareness and remove the utf8 package. in the v3 roadmap and therefore does not relate to this PR specifically.

@bitinn
Copy link
Collaborator

bitinn commented Nov 18, 2019

@Richienb it's also failing in a test for data uri support.

should accept data uri of plain text

https://travis-ci.org/bitinn/node-fetch/jobs/613306983

@bitinn
Copy link
Collaborator

bitinn commented Nov 18, 2019

But I guess due to linting issues no PR to v3 will pass? Or was the linting test optional?

@bitinn
Copy link
Collaborator

bitinn commented Nov 18, 2019

I would say we could have a test for using fetch-charset-detection in our test suite? It's not strictly necessary for unit testing, but nice to confirm it works integration-wise.

@Richienb
Copy link
Member Author

@bitinn

it's also failing in a test for data uri support.

That's not related to this PR.

But I guess due to linting issues no PR to v3 will pass? Or was the linting test optional?

It looks like VSCode autoformatted the code incorrectly. I'll run yarn lint --fix and push the changes.

I would say we could have a test for using fetch-charset-detection in our test suite? It's not strictly necessary for unit testing, but nice to confirm it works integration-wise.

https://github.com/Richienb/fetch-charset-detection/blob/master/test.js

@Richienb
Copy link
Member Author

Richienb commented Nov 18, 2019

@bitinn The 3.x branch managed to get some bad VSCode formatting. However, the build for this branch specifically hasn't. I've fixed it in a0a1200.

@bitinn
Copy link
Collaborator

bitinn commented Nov 18, 2019

then it's good to go.

@Richienb
Copy link
Member Author

All reviewers approve. Time to merge!

@Richienb Richienb merged commit af70087 into 3.x Nov 23, 2019
@Richienb Richienb deleted the add-fetch-charset-detection branch November 23, 2019 21:32
bitinn added a commit that referenced this pull request Mar 13, 2020
* feat: Migrate TypeScript types (#669)

* style: Introduce linting via XO

* fix: Fix tests

* chore!: Drop support for nodejs 4 and 6

* chore: Fix Travis CI yml

* Use old Babel (needs migration)

* chore: lint everything

* chore: Migrate to microbundle

* Default response.statusText should be blank (#578)

* fix: Use correct AbortionError message

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Use modern @babel/register

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Remove redundant packages

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Readd form-data

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* fix: Fix tests and force utf8-encoded urls

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* lint index.js

* Update devDependencies & ignore `test` directory in linter options

* Remove unnecessary eslint-ignore comment

* Update the `lint` script to run linter on every file

* Remove unused const & unnecessary import

* TypeScript: Fix Body.blob() wrong type (DefinitelyTyped/DefinitelyTyped#33721)

* chore: Lint as part of the build process

* fix: Convert Content-Encoding to lowercase (#672)

* fix: Better object checks (#673)

* Fix stream piping (#670)

* chore: Remove useless check

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* style: Fix lint

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* style: Fix lint

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* refactor: Modernise code

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Ensure all files are properly included

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Update deps and utf8 should be in dependencies

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* test: Drop Node v4 from tests

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* test: Modernise code

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Move errors to seperate directory

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* refactor: Add fetch-blob (#678)

* feat: Migrate data uri integration

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Allow setting custom highWaterMark via node-fetch options (#386) (#671)

* Expose highWaterMark option to body clone function

* Add highWaterMark to responseOptions

* Add highWaterMark as node-fetch-only option

* a way to silently pass highWaterMark to clone

* Chai helper

* Server helper

* Tests

* Remove debug comments

* Document highWaterMark option

* Add TypeScript types for the new highWaterMark option

* feat: Include system error in FetchError if one occurs (#654)

* style: Add editorconfig

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore!: Drop NodeJS v8

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Remove legacy code for node < 8

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Use proper checks for ArrayBuffer and AbortError

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Use explicitly set error name in checks

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Propagate size and timeout to cloned response (#664)

* Remove --save option as it isn't required anymore (#581)

* Propagate size and timeout to cloned response


Co-authored-by: Steve Moser <contact@stevemoser.org>
Co-authored-by: Antoni Kepinski <xxczaki@pm.me>

* Update Response types

* Update devDependencies

* feat: Fallback to blob type (Closes: #607)

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* style: Update formatting

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* style: Fix linting issues

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: Add info on patching the global object

* docs: Added non-globalThis polyfill

* Replace deprecated `url.resolve` with the new WHATWG URL

* Update devDependencies

* Format code in examples to use `xo` style

* Verify examples with RunKit and edit them if necessary

* Add information about TypeScript support

* Document the new `highWaterMark` option

* Add Discord badge & information about Open Collective

* Style change

* Edit acknowledgement & add "Team" section

* fix table

* Format example code to use xo style

* chore: v3 release changelog

* Add the recommended way to fix `highWaterMark` issues

* docs: Add simple Runkit example

* fix: Properly set the name of the errors.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: Add AbortError to documented types

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: AbortError proper typing parameters

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: Add example code for Runkit

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Replace microbundle with @pika/pack (#689)

* gitignore the pkg/ directory

* Move TypeScript types to the root of the project

* Replace microbundle with @pika/pack

* chore: Remove @pika/plugin-build-web and revert ./dist output directory

Signed-off-by: Richie Bendall <richiebendall@gmail.com>


Co-authored-by: Richie Bendall <richiebendall@gmail.com>

* fix incorrect statement in changelog

* chore: v3.x upgrade guide

* Change the Open Collective button

* docs: Encode support button as Markdown instead of HTML

* chore: Ignore proper directory in xo

* Add an "Upgrading" section to readme

* Split the upgrade guide into 2 files & add the missing changes about v3.x

* style: Lint test and example files

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Move *.md files to the `docs` folder (except README.md)

* Update references to files

* Split LIMITS.md into 2 files (as of v2.x and v3.x)

* chore: Remove logging statement

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* style: Fix lint

* docs: Correct typings for systemError in FetchError (Fixes #697)

* refactor: Replace `encoding` with `fetch-charset-detection`. (#694)

* refactor: Replace `encoding` with `fetch-charset-detection`.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* refactor: Move writing to stream back to body.js

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* refactor: Only put convertBody in fetch-charset-detection and refactor others.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* test: Readd tests for getTotalBytes and extractContentType

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Revert package.json indention

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Remove optional dependency

* docs: Replace code for fetch-charset-detection with documentation.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Remove iconv-lite

* fix: Use default export instead of named export for convertBody

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Remove unneeded installation of fetch-charset-detection in the build

* docs: Fix typo

* fix: Throw SyntaxError instead of FetchError in case of invalid… (#700)

* fix: Throw SyntaxError instead of FetchError in case of invalid JSON

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: Add to upgrade guide

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Remove deprecated url.parse from test

* Remove deprecated url.parse from server

* fix: Proper data uri to buffer conversion (#703)

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Add funding info

* fix: Flawed property existence test (#706)

Fix a problem where not all prototype methods are copied from the Body via the mixin method due to a failure to properly detect properties in the target. The current code uses the `in` operator, which may return properties lower down the inheritance chain, thus causing them to fail the copy. The new code properly calls the `.hasOwnProperty()` method to make the determination.

* fix: Properly handle stream pipeline double-fire

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: Fix spelling

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Add `funding` field to package.json (#708)

* Fix: Do not set ContentLength to NaN (#709)

* do not set ContentLength to NaN

* lint

* docs: Add logo

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Update repository name from bitinn/node-fetch to node-fetch/node-fetch.

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Fix unit tests

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore(deps): Bump @pika/plugin-copy-assets from 0.7.1 to 0.8.1 (#713)

Bumps [@pika/plugin-copy-assets](https://github.com/pikapkg/builders) from 0.7.1 to 0.8.1.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.7.1...v0.8.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* chore(deps): Bump @pika/plugin-build-types from 0.7.1 to 0.8.1 (#710)

Bumps [@pika/plugin-build-types](https://github.com/pikapkg/builders) from 0.7.1 to 0.8.1.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.7.1...v0.8.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump nyc from 14.1.1 to 15.0.0 (#714)

Bumps [nyc](https://github.com/istanbuljs/nyc) from 14.1.1 to 15.0.0.
- [Release notes](https://github.com/istanbuljs/nyc/releases)
- [Changelog](https://github.com/istanbuljs/nyc/blob/master/CHANGELOG.md)
- [Commits](istanbuljs/nyc@v14.1.1...v15.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* chore: Update travis ci url

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore(deps): Bump mocha from 6.2.2 to 7.0.0 (#711)

Bumps [mocha](https://github.com/mochajs/mocha) from 6.2.2 to 7.0.0.
- [Release notes](https://github.com/mochajs/mocha/releases)
- [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
- [Commits](mochajs/mocha@v6.2.2...v7.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* feat: Allow excluding a user agent in a fetch request by setting… (#715)

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* Bump @pika/plugin-build-node from 0.7.1 to 0.8.1 (#717)

Bumps [@pika/plugin-build-node](https://github.com/pikapkg/builders) from 0.7.1 to 0.8.1.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.7.1...v0.8.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump @pika/plugin-standard-pkg from 0.7.1 to 0.8.1 (#716)

Bumps [@pika/plugin-standard-pkg](https://github.com/pikapkg/builders) from 0.7.1 to 0.8.1.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.7.1...v0.8.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump form-data from 2.5.1 to 3.0.0 (#712)

Bumps [form-data](https://github.com/form-data/form-data) from 2.5.1 to 3.0.0.
- [Release notes](https://github.com/form-data/form-data/releases)
- [Commits](form-data/form-data@v2.5.1...v3.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* fix: typo

* update suggestion

* feat: Added missing redirect function (#718)

* added missing redirect function
* chore: Add types

Co-authored-by: Richie Bendall <richiebendall@gmail.com>

* fix: Use req.setTimeout for timeout (#719)

* chore: Update typings comment

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* chore: Update deps

Signed-off-by: Richie Bendall <richiebendall@gmail.com>

* docs: center badges & Open Collective button

* docs: add missing comma

* Remove current stable & LTS node version numbers from the comments

I don't think we really want to update them

* Bump xo from 0.25.4 to 0.26.1 (#730)

Bumps [xo](https://github.com/xojs/xo) from 0.25.4 to 0.26.1.
- [Release notes](https://github.com/xojs/xo/releases)
- [Commits](xojs/xo@v0.25.4...v0.26.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump @pika/plugin-build-types from 0.8.3 to 0.9.2 (#729)

Bumps [@pika/plugin-build-types](https://github.com/pikapkg/builders) from 0.8.3 to 0.9.2.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.8.3...v0.9.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump @pika/plugin-standard-pkg from 0.8.3 to 0.9.2 (#726)

Bumps [@pika/plugin-standard-pkg](https://github.com/pikapkg/builders) from 0.8.3 to 0.9.2.
- [Release notes](https://github.com/pikapkg/builders/releases)
- [Commits](FredKSchott/pika-pack-builders@v0.8.3...v0.9.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* docs: Update information about `req.body` type in v3.x release

* Add information about removed body type to the v3 upgrade guide

* add awesome badge

* Show 2 ways of importing node-fetch (CommonJS & ES module)

* update dependencies

* lint

* refactor: Replace `url.parse` with `new URL()` (#701)

* chore: replace `url.parse` with `new URL()`

* lint

* handle relative URLs

* Change error message

* detect whether the url is absolute or not

* update tests

* drop relative url support

* lint

* fix tests

* typo

* Add information about dropped arbitrary URL support in v3.x upgrade guide

* set xo linting rule (node/no-deprecated-api) to on

* remove the `utf8` dependency

* fix

* refactor: split tests into several files, create the `utils` directory

* Update package.json scripts & remove unnecessary xo linting rules

* refactor: turn on some xo linting rules to improve code quality

* fix tests

* Remove invalid urls

* fix merge conflict

* update the upgrade guide

* test if URLs are encoded as UTF-8

* update xo to 0.28.0

* chore: Build before publishing

* v3.0.0-beta.1

* fix lint on test/main.js

Co-authored-by: Richie Bendall <richiebendall@gmail.com>
Co-authored-by: Antoni Kepinski <xxczaki@pm.me>
Co-authored-by: aeb-sia <50743092+aeb-sia@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Steve Moser <contact@stevemoser.org>
Co-authored-by: Erick Calder <e@arix.com>
Co-authored-by: Yaacov Rydzinski <yaacovCR@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Jimmy Wärting <jimmy@warting.se>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants