Skip to content

flake8, bugbear, pyupgrade → ruff#12002

Merged
jacobtomlinson merged 6 commits intodask:mainfrom
DimitriPapadopoulos:ruff
Oct 31, 2025
Merged

flake8, bugbear, pyupgrade → ruff#12002
jacobtomlinson merged 6 commits intodask:mainfrom
DimitriPapadopoulos:ruff

Conversation

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor

@DimitriPapadopoulos DimitriPapadopoulos commented Jul 7, 2025

I plan on moving from isort to rule set I and from black to ruff format (if that's indeed what you want) in different PRs.

@DimitriPapadopoulos DimitriPapadopoulos force-pushed the ruff branch 3 times, most recently from 6214b7c to c3a941f Compare July 7, 2025 20:15
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 7, 2025

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

      9 files  ±0        9 suites  ±0   3h 6m 6s ⏱️ - 2m 9s
 18 139 tests ±0   16 924 ✅ ±0   1 215 💤 ±0  0 ❌ ±0 
162 479 runs  ±0  150 386 ✅ +1  12 093 💤  - 1  0 ❌ ±0 

Results for commit b172c5f7. ± Comparison against base commit a9d0e52.

♻️ This comment has been updated with latest results.

@DimitriPapadopoulos DimitriPapadopoulos force-pushed the ruff branch 3 times, most recently from 528a8d5 to ec55aa1 Compare July 9, 2025 19:41
@github-actions github-actions bot added the needs attention It's been a while since this was pushed on. Needs attention from the owner or a maintainer. label Oct 6, 2025
@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

Done. Note that UP038 has been deprecated end eventually removed. Ignoring it now emits a warning.

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

This is a new issue. Is it caused by this PR?

FAILED dask/bytes/tests/test_http.py::test_parquet[pyarrow] - pyarrow.lib.ArrowInvalid: Invalid number of indices: 0

The remote file nation.impala.parquet appears to be available:

@pytest.mark.network
@pytest.mark.parametrize(
"engine",
["pyarrow"],
)
def test_parquet(engine):
pytest.importorskip("requests", minversion="2.21.0")
dd = pytest.importorskip("dask.dataframe")
pytest.importorskip(engine)
df = dd.read_parquet(
[
"https://github.com/Parquet/parquet-compatibility/raw/"
"master/parquet-testdata/impala/1.1.1-NONE/"
"nation.impala.parquet"
],
engine=engine,
).compute()
assert df.n_nationkey.tolist() == list(range(25))
assert df.columns.tolist() == ["n_nationkey", "n_name", "n_regionkey", "n_comment"]

Copy link
Copy Markdown
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is looking good now thanks. I note that CI is passing on main, and failing here, which suggests failures are related to the changes in here. Could you take a look?

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

@jacobtomlinson I may be missing something here, I don't see how the failure could be related to the changes in this PR. I haven't changed actual code, just linting tools and linting directives. This is certainly caused by the update of a dependency, but I cannot identify which one. Time permitting, I'll download logs from both branches and diff them to find the culprit.

@DimitriPapadopoulos DimitriPapadopoulos force-pushed the ruff branch 2 times, most recently from 3809468 to 4a36165 Compare October 30, 2025 21:07
@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

After rebasing, most tests pass. Some transient error probably caused previous failures. The only test that fails shows a new error, probably some "random" error again. Could you please restart the failing test? Eventually it should pass.

I think the tests have become fragile for some reason, and fail "randomly". I don't know enough about this package to fix this fragility, it's probably related to multiprocessing.

@jacobtomlinson
Copy link
Copy Markdown
Member

Based on other PRs that have been raised recently I've not found CI to be very flaky, so I'm concerned that this PR is behaving this way. I agree that the changes here should just be format/lint related and not change actual functionality. But the fact the CI keeps failing in these ways makes me nervous that there is in fact some functional change in here that we've missed.

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

I know, but I've looked again and again and I only see changed # noqa directives and associated comments in the Python source code. The only "change" in actual code is a mere formatting change:

-         raise ValueError(
-             f"Index column {index_name} not found in statistics"  # noqa: E713
-         )
+         raise ValueError(f"Index column {index_name} not found in statistics")

The only possible effect I can imagine is that ruff runs much faster than flake8/pyupgrade, and this has an effect on tests running in parallel or after the linter somehow - but again the failure would result from a preexisting race condition in other tests.

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

Also note that the error is stochastic, as shown by the success of the new run of the previously failing test. This does seem to point to some sort of race condition.

@DimitriPapadopoulos
Copy link
Copy Markdown
Contributor Author

DimitriPapadopoulos commented Oct 31, 2025

I see you've been able to identify the initial error as an effect of #47981.

How about I rebase one last time onto your related fix #12124 and see how it goes?

DimitriPapadopoulos and others added 2 commits October 31, 2025 14:54
Co-authored-by: Jacob Tomlinson <jacobtomlinson@users.noreply.github.com>
Copy link
Copy Markdown
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I've gone over the diff again and can't see anything that causes concern.

@jacobtomlinson jacobtomlinson merged commit 17a245a into dask:main Oct 31, 2025
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs attention It's been a while since this was pushed on. Needs attention from the owner or a maintainer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace flake8 with ruff

2 participants