Fix compatibility with Python 3.10 by simonw · Pull Request #1481 · simonw/datasette

simonw · 2021-10-07T20:34:23Z

No description provided.

codecov · 2021-10-07T20:42:37Z

Codecov Report

Merging #1481 (77542e7) into main (6388617) will decrease coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results

@@            Coverage Diff             @@
##             main    #1481      +/-   ##
==========================================
- Coverage   91.83%   91.82%   -0.02%     
==========================================
  Files          34       34              
  Lines        4421     4426       +5     
==========================================
+ Hits         4060     4064       +4     
- Misses        361      362       +1

Impacted Files	Coverage Δ
datasette/app.py	`95.36% <0.00%> (-0.14%)`	⬇️
datasette/views/base.py	`95.41% <0.00%> (ø)`
datasette/views/database.py	`97.56% <0.00%> (ø)`
datasette/views/table.py	`96.00% <0.00%> (+<0.01%)`	⬆️
datasette/utils/__init__.py	`94.78% <0.00%> (+0.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6388617...50005bd. Read the comment docs.

simonw · 2021-10-07T20:44:43Z

The 3.10 tests failed a lot. Trying to run this locally:

/tmp % pyenv install 3.10
python-build: definition not found: 3.10

The following versions contain `3.10' in the name:
  3.10.0a6
  3.10-dev
  miniconda-3.10.1
  miniconda3-3.10.1

See all available versions with `pyenv install --list'.

If the version you need is missing, try upgrading pyenv:

  brew update && brew upgrade pyenv

So trying:

brew update && brew upgrade pyenv

Then did this:

/tmp % brew upgrade pyenv 
==> Upgrading 1 outdated package:
pyenv 1.2.24.1 -> 2.1.0

This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

But it looks like I have 3.10.0 available to pyenv now.

/tmp % pyenv install 3.10.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.0.tar.xz...
-> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
Installing Python-3.10.0...
...

simonw · 2021-10-08T19:40:23Z

Then I created myself a temporary 3.10 environment using pipenv like so:

cd /tmp
mkdir py310
cd py310
pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

And used that with my Datasette checkout like so:

cd ~/.../datasette
pip install -e '.[test]'
pytest

simonw · 2021-10-08T19:42:00Z

Running pytest -x --pdb helped me see this error:

  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request
    await self.ds.refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 344, in refresh_schemas
    await self._refresh_schemas()
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 349, in _refresh_schemas
    await init_internal_db(internal_db)
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py", line 5, in init_internal_db
    await db.execute_write(
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 102, in execute_write
    return await self.execute_write_fn(_inner, block=block)
  File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 113, in execute_write_fn
    reply_queue = janus.Queue()
  File "/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py", line 39, in __init__
    self._async_not_empty = asyncio.Condition(self._async_mutex)
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py", line 234, in __init__
    raise ValueError("loop argument must agree with lock")
ValueError: loop argument must agree with lock

simonw · 2021-10-08T19:43:33Z

So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.

simonw · 2021-10-08T19:47:29Z

Only mention I can find of that "loop argument must agree with lock" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278

simonw · 2021-10-08T19:49:08Z

Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
    """Asynchronous equivalent to threading.Condition.
    This class implements condition variable objects. A condition variable
    allows one or more coroutines to wait until they are notified by another
    coroutine.
    A new Lock object is created and used as the underlying lock.
    """

    def __init__(self, lock=None, *, loop=mixins._marker):
        super().__init__(loop=loop)
        if lock is None:
            lock = Lock()
        elif lock._loop is not self._get_loop():
            raise ValueError("loop argument must agree with lock")

simonw · 2021-10-08T19:50:52Z

And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42

class Queue(Generic[T]):
    def __init__(self, maxsize: int = 0) -> None:
        self._loop = current_loop()
        self._maxsize = maxsize

        self._init(maxsize)

        self._unfinished_tasks = 0

        self._sync_mutex = threading.Lock()
        self._sync_not_empty = threading.Condition(self._sync_mutex)
        self._sync_not_full = threading.Condition(self._sync_mutex)
        self._all_tasks_done = threading.Condition(self._sync_mutex)

        self._async_mutex = asyncio.Lock()
        # "loop argument must agree with lock" exception is raised here:
        self._async_not_empty = asyncio.Condition(self._async_mutex)
        self._async_not_full = asyncio.Condition(self._async_mutex)
        self._finished = asyncio.Event()
        self._finished.set()

simonw · 2021-10-08T20:33:42Z

There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a "loop argument must agree with lock" exception.

This code raises that exception:
asyncio.Condition(asyncio.Lock())
This worked in previous Python versions.

Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:
import asyncio

# This runs without an exception:
print(asyncio.Condition(asyncio.Lock()))

# This does not work:
async def example():
    print(asyncio.Condition(asyncio.Lock()))

# This raises "ValueError: loop argument must agree with lock":
asyncio.run(example())

simonw · 2021-10-08T23:41:39Z

I submitted a PR to Janus with a workaround for this: aio-libs/janus#359

simonw · 2021-10-09T00:04:54Z

I applied my PR against Janus to my local copy of Datasette like so:

pip uninstall janus
pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

Then I ran the Datasette tests and got a much happier pass rate.

simonw · 2021-10-16T19:07:38Z

This is blocking an upgrade for the Homebrew Datasette package: Homebrew/homebrew-core#86932

simonw · 2021-10-16T23:19:51Z

Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.

Refs #1482 Refs aio-libs/janus#358

This reverts commit 98dcabc.

Closes #1486

…1485) Updates the requirements on [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-timeout/releases) - [Commits](https://github.com/pytest-dev/pytest-timeout/commits) --- updated-dependencies: - dependency-name: pytest-timeout dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-xdist/releases) - [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest-xdist@v2.2.1...v2.4.0) --- updated-dependencies: - dependency-name: pytest-xdist dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Change "above" to "below" to correct correspondence of reference to example.

Updates the requirements on [pluggy](https://github.com/pytest-dev/pluggy) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pluggy/releases) - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@0.13.0...1.0.0) --- updated-dependencies: - dependency-name: pluggy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [black](https://github.com/psf/black) from 21.7b0 to 21.9b0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/commits) --- updated-dependencies: - dependency-name: black dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Updates the requirements on [beautifulsoup4](http://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version. --- updated-dependencies: - dependency-name: beautifulsoup4 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Upgrade to httpx 0.20, closes #1488 * TestClient.post() should not default to following redirects

Updates the requirements on [pyyaml](https://github.com/yaml/pyyaml) to permit the latest version. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](yaml/pyyaml@5.3...6.0) --- updated-dependencies: - dependency-name: pyyaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Rework the `--static` documentation to better differentiate between the filesystem and serving locations. Closes #1457 Co-authored-by: Simon Willison <swillison@gmail.com>

Refs #1467, refs simonw/datasette-auth-tokens#4

Refs #942, #1404, #1405, #1416, #1420, #1421, #1422, #1423, #1425, #1431, #1443, #1446, #1449, #1467, #1469, #1470, #1488

Try running tests against Python 3.10

18a4907

simonw changed the title ~~Try running tests against Python 3.10~~ Add support for Python 3.10 Oct 8, 2021

simonw mentioned this pull request Oct 8, 2021

Error with Python 3.10: "ValueError: loop argument must agree with lock" aio-libs/janus#358

Closed

simonw changed the title ~~Add support for Python 3.10~~ Fix compatibility with Python 3.10 Oct 8, 2021

simonw mentioned this pull request Oct 9, 2021

Support Python 3.10 #1482

Closed

follower mentioned this pull request Oct 17, 2021

Gratuitous test. follower/experiment-github-issue-drafts#7

Closed

simonw and others added 11 commits October 24, 2021 15:04

Upgrade to Janus 0.6.2 for Python 3.10

77542e7

Refs #1482 Refs aio-libs/janus#358

asyncio_run helper to deal with a 3.10 warning, refs #1482

94b0aa9

Update test to handle Python 3.10 error message differenc, refs #1482

31c49e6

Fix bug with ?_next=x&_sort=rowid, closes #1470

4486018

Revert "asyncio_run helper to deal with a 3.10 warning, refs #1482"

8d1ec24

This reverts commit 98dcabc.

Don't persist _next in hidden field, closes #1483

f70b63c

Added instructions for installing plugins via pipx

1fc1d40

Closes #1486

Update full_text_search.rst (#1474)

2272c98

Change "above" to "below" to correct correspondence of reference to example.

simonw and others added 14 commits October 24, 2021 15:11

Fix for cog menu default facet bug, closes #1469

d255eb5

Upgrade to httpx 0.20

a3551c6

* Upgrade to httpx 0.20, closes #1488 * TestClient.post() should not default to following redirects

Rework the --static documentation

27fefad

Rework the `--static` documentation to better differentiate between the filesystem and serving locations. Closes #1457 Co-authored-by: Simon Willison <swillison@gmail.com>

--cors Access-Control-Allow-Headers: Authorization

27f326b

Refs #1467, refs simonw/datasette-auth-tokens#4

Updated --cors documentation, refs #1467

40265f3

Release 0.59

c2c9957

Refs #942, #1404, #1405, #1416, #1420, #1421, #1422, #1423, #1425, #1431, #1443, #1446, #1449, #1467, #1469, #1470, #1488

Fixed typo in release notes

bcefca8

Fixed typo in release notes

dd46292

Unwrapped some documentation text

b3a9879

Docs on named parameters with cast as real/integer, closes #1496

52a88d0

Add 3.10 to classifiers

50005bd

simonw merged commit 96a823f into main Oct 24, 2021

simonw deleted the python-3-10 branch October 24, 2021 22:19

Uh oh!

Conversation

simonw commented Oct 7, 2021

Uh oh!

codecov bot commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

simonw commented Oct 7, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 8, 2021

Uh oh!

simonw commented Oct 9, 2021

Uh oh!

simonw commented Oct 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonw commented Oct 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Oct 7, 2021 •

edited

Loading

simonw commented Oct 16, 2021 •

edited

Loading