[BUG] Can't access a page that is using zstd

NOTE: IF YOU'RE UNSURE, PLEASE LOOK AT OR OPEN A DISCUSSION INSTEAD! https://github.com/mborsetti/webchanges/discussions


**Describe the bug.**
A clear and concise description of what the bug is.  If it's related to an error, please also attach the traceback 
of the exception obtained by running ``webchanges`` with ``-v`` or ``-vv``.
```
2025-03-26 14:50:28,028 cli[123378913575168] INFO: webchanges: 3.29.0 Copyright 2020- Mike Borsetti
2025-03-26 14:50:28,028 cli[123378913575168] INFO: CPython: 3.13.2 ('main', 'Mar 13 2025 14:29:07') GCC 14.2.0
2025-03-26 14:50:28,029 cli[123378913575168] INFO: System: Linux-6.14.0-11-generic-x86_64-with-glibc2.41
/home/mhalano/.local/lib/python3.13/site-packages/webchanges/cli.py:248: RuntimeWarning: Hooks file not imported because /home/mhalano/.config/webchanges/hooks.py is not a file
  warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
2025-03-26 14:50:28,245 storage[123378913575168] INFO: The configuration file /home/mhalano/.config/webchanges/config.yaml is missing directive(s); the following default values are being used:
database:
  engine: sqlite3
  max_snapshots: 4
differ_defaults:
  _note: Default directives that are applied to individual differs.
  ai_google: {}
  command: {}
  deepdiff: {}
  image: {}
  table: {}
  unified: {}
  wdiff: {}
display:
  empty-diff: false
footnote: null
report:
  browser:
    enabled: false
  discord:
    colored: true
    embed: true
    enabled: false
    max_message_length: null
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
    webhook_url: ''
  email:
    sendmail:
      path: sendmail
  gotify:
    enabled: false
    priority: 0
    server_url: ''
    title: ''
    token: ''
  html:
    footer: true
    separate: false
    title: '[webchanges] {count} changes{jobs_files}: {jobs}'
  ifttt:
    enabled: false
    event: ''
    key: ''
  mailgun:
    api_key: ''
    domain: ''
    enabled: false
    from_mail: ''
    from_name: ''
    region: us
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
    to: ''
  markdown:
    details: true
    footer: true
    minimal: false
    separate: false
  matrix:
    access_token: ''
    enabled: false
    homeserver: ''
    room_id: ''
  prowl:
    api_key: ''
    application: ''
    enabled: false
    priority: 0
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
  pushbullet:
    api_key: ''
    enabled: false
  pushover:
    app: ''
    device: null
    enabled: false
    priority: normal
    sound: spacealarm
    user: ''
  run_command:
    command: ''
    enabled: false
  telegram:
    bot_token: ''
    chat_id: ''
    enabled: false
    silent: false
  text:
    separate: false
  tz: null
  webhook:
    enabled: false
    markdown: false
    max_message_length: null
    rich_text: null
    webhook_url: ''
  xmpp:
    enabled: false
    insecure_password: ''
    recipient: ''
    sender: ''
See documentation at https://webchanges.readthedocs.io/en/stable/configuration.html
2025-03-26 14:50:28,245 storage[123378913575168] INFO: Loaded configuration from /home/mhalano/.config/webchanges/config.yaml
2025-03-26 14:50:28,245 storage[123378913575168] INFO: Using sqlite3 3.46.1 database at /home/mhalano/.local/share/webchanges/snapshots.db
2025-03-26 14:50:28,247 jobs[123378913575168] INFO: Job 1: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,248 jobs[123378913575168] INFO: Job 2: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,248 jobs[123378913575168] INFO: Job 3: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,249 jobs[123378913575168] INFO: Job 4: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,249 jobs[123378913575168] INFO: Job 5: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,250 jobs[123378913575168] INFO: Job 6: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,251 jobs[123378913575168] INFO: Job 7: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,251 jobs[123378913575168] INFO: Job 8: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,252 jobs[123378913575168] INFO: Job 9: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,252 jobs[123378913575168] INFO: Job 10: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,253 jobs[123378913575168] INFO: Job 11: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,254 jobs[123378913575168] INFO: Job 12: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,254 jobs[123378913575168] INFO: Job 13: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,255 jobs[123378913575168] INFO: Job 14: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,256 jobs[123378913575168] INFO: Job 15: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,256 jobs[123378913575168] INFO: Job 16: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,257 jobs[123378913575168] INFO: Job 17: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,257 jobs[123378913575168] INFO: Job 18: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,258 jobs[123378913575168] INFO: Job 19: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,258 jobs[123378913575168] INFO: Job 20: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,259 jobs[123378913575168] INFO: Job 21: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,260 jobs[123378913575168] INFO: Job 22: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,260 jobs[123378913575168] INFO: Job 23: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,262 jobs[123378913575168] INFO: Job 28: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,263 jobs[123378913575168] INFO: Job 29: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,264 jobs[123378913575168] INFO: Job 31: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,264 jobs[123378913575168] INFO: Job 32: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,265 jobs[123378913575168] INFO: Job 33: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,265 jobs[123378913575168] INFO: Job 34: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,266 jobs[123378913575168] INFO: Job 35: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,267 jobs[123378913575168] INFO: Job 36: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,267 jobs[123378913575168] INFO: Job 37: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,268 jobs[123378913575168] INFO: Job 38: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,268 jobs[123378913575168] INFO: Job 39: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,269 jobs[123378913575168] INFO: Job 40: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,269 jobs[123378913575168] INFO: Job 41: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,270 jobs[123378913575168] INFO: Job 42: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,270 jobs[123378913575168] INFO: Job 45: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,271 jobs[123378913575168] INFO: Job 46: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,271 jobs[123378913575168] INFO: Job 47: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,272 jobs[123378913575168] INFO: Job 48: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,272 jobs[123378913575168] INFO: Job 49: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,273 jobs[123378913575168] INFO: Job 50: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,274 jobs[123378913575168] INFO: Job 51: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,274 jobs[123378913575168] INFO: Job 52: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,275 jobs[123378913575168] INFO: Job 53: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,276 jobs[123378913575168] INFO: Job 54: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,277 jobs[123378913575168] INFO: Job 55: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,278 jobs[123378913575168] INFO: Job 56: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,279 jobs[123378913575168] INFO: Job 57: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,280 jobs[123378913575168] INFO: Job 58: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,280 jobs[123378913575168] INFO: Job 59: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,281 jobs[123378913575168] INFO: Job 60: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,281 jobs[123378913575168] INFO: Job 61: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,282 jobs[123378913575168] INFO: Job 62: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,282 jobs[123378913575168] INFO: Job 63: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,283 jobs[123378913575168] INFO: Job 64: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,283 jobs[123378913575168] INFO: Job 65: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,284 jobs[123378913575168] INFO: Job 66: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,284 jobs[123378913575168] INFO: Job 67: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,285 jobs[123378913575168] INFO: Job 68: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,286 jobs[123378913575168] INFO: Job 69: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,286 jobs[123378913575168] INFO: Job 70: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,287 jobs[123378913575168] INFO: Job 71: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,287 jobs[123378913575168] INFO: Job 72: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,288 jobs[123378913575168] INFO: Job 73: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,288 storage[123378913575168] INFO: Loaded 73 jobs from /home/mhalano/.config/webchanges/jobs.yaml
2025-03-26 14:50:28,288 handler[123378913575168] INFO: Job 72: https://help.netflix.com/en/node/30081 started processing (UrlJob)
2025-03-26 14:50:28,288 jobs[123378913575168] INFO: Job 72: Sending GET request to https://help.netflix.com/en/node/30081
2025-03-26 14:50:28,289 jobs[123378913575168] INFO: Job 72: Using the HTTPX HTTP client library with HTTP/2 support
2025-03-26 14:50:29,165 _client[123378913575168] INFO: HTTP Request: GET https://help.netflix.com/en/node/30081 "HTTP/2 200 OK"
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: Job ended with error; incrementing cumulative error runs to 3
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: https://help.netflix.com/en/node/30081 ended processing
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: Got exception while processing job <url url='https://help.netflix.com/en/node/30081' headers={} ignore_cached=True method='GET' ignore_connection_errors=True ignore_http_error_codes='4xx, 5xx' ignore_timeout_errors=True filters=[{'xpath': '/html/body/div[1]/div[4]/div/div[2]/div/div[1]/section/form[6]/div/div/div[2]/table/tbody'}, {'html2text': None}, {'striplines': None}] index_number=72 name="Changes on Netflix's resolution on Linux"
Traceback (most recent call last):
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_decoders.py", line 185, in decode
    output.write(self.decompressor.decompress(data))
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
zstd.ZstdError: cannot use a decompressobj multiple times

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/handler.py", line 231, in process
    data, self.new_etag, mime_type = self.job.retrieve(self, headless)
                                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/jobs.py", line 1066, in retrieve
    data, etag, mime_type = self._retrieve_httpx(headers=headers, timeout=timeout)
                            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/jobs.py", line 773, in _retrieve_httpx
    response = client.request(
        method=self.method,  # type: ignore[arg-type]
    ...<2 lines>...
        params=self.params,
    )
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 928, in send
    raise exc
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 922, in send
    response.read()
    ~~~~~~~~~~~~~^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_models.py", line 881, in read
    self._content = b"".join(self.iter_bytes())
                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_models.py", line 898, in iter_bytes
    decoded = decoder.decode(raw_bytes)
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_decoders.py", line 191, in decode
    raise DecodingError(str(exc)) from exc
httpx.DecodingError: cannot use a decompressobj multiple times
2025-03-26 14:50:29,214 reporters[123378913575168] INFO: Submitting with stdout (<class 'webchanges.reporters.StdoutReporter'>)
2025-03-26 14:50:29,223 command[123378913575168] INFO: Exiting with exit code 0
===========================================================================
01. ERROR: Changes on Netflix's resolution on Linux
===========================================================================

---------------------------------------------------------------------------
ERROR: Changes on Netflix's resolution on Linux (https://help.netflix.com/en/node/30081)
---------------------------------------------------------------------------

cannot use a decompressobj multiple times
---------------------------------------------------------------------------


--
Checked 1 source in 1.1 seconds with webchanges 3.29.0.
```

**Steps to reproduce.**
Include relevant snippets of anything required (e.g. jobs and configuration files) for anyone to fully reproduce the 
bug.
Try to retrieve the Netflix's URL.

**Expected behavior.**
A clear and concise description of what you expected to happen.
Should return some values.

**Screen scrape/screenshots.**
If applicable, add screen scrape or screenshots to help explain the bug; please use ``-v`` or ``-vv`` to capture 
logging.

**Version info.**
Please run ``webchanges --detailed-versions`` and paste the output here.
```
➜ webchanges --detailed-versions
/home/mhalano/.local/lib/python3.13/site-packages/webchanges/cli.py:248: RuntimeWarning: Hooks file not imported because /home/mhalano/.config/webchanges/hooks.py is not a file
  warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
Software:
• webchanges: 3.29.0
• CPython: 3.13.2 ('main', 'Mar 13 2025 14:29:07') GCC 14.2.0
• SQLite: 3.46.1

System:
• Platform: Linux-6.14.0-11-generic-x86_64-with-glibc2.41, x86_64
• Processor: x86_64
• CPUs (logical): 16
• Free memory: 43.5G physical plus 0.0B swap.
• Free disk '/': 201.8G (45.4%)
• --max-threads default: 20

Installed PyPi dependencies:
• cssselect: 1.3.0
• h2: 4.2.0
• html2text: 2024.2.26
• httpx: 0.28.1
  - anyio: 4.9.0
  - certifi: 2025.1.31
  - idna: 3.10
• markdown2: 2.5.3
• msgpack: 1.1.0
• platformdirs: 4.3.7
• pyyaml: 6.0.2
• zstandard: 0.23.0

Installed dpkg dependencies:
• jq
   - jq: 1.7.1-3build1
• Pillow
   - libjpeg-dev: 8c-2ubuntu11
   - zlib1g-dev: 1:1.3.dfsg+really1.3.1-1ubuntu1
   - libtiff-dev: 4.5.1+git230720-4ubuntu4
   - libfreetype-dev: 2.13.3+dfsg-1
   - libwebp-dev: 1.5.0-0.1
   - libimagequant-dev: 2.18.0-1build1
   - libraqm-dev: 0.10.2-1
   - libxcb1-dev: 1.17.0-2
```

**Additional context.**
Add any other context about the problem.


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Can't access a page that is using zstd #98

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[BUG] Can't access a page that is using zstd #98

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions