Skip to content

scandir returns Stats relative to its directory.#19246

Merged
stuhood merged 2 commits intomainfrom
stuhood/relative-scandir
Jun 5, 2023
Merged

scandir returns Stats relative to its directory.#19246
stuhood merged 2 commits intomainfrom
stuhood/relative-scandir

Conversation

@stuhood
Copy link
Member

@stuhood stuhood commented Jun 4, 2023

The scandir node (which currently holds Stat entries relative to the build root) was 28% of memory usage for one use case on #19053.

This change moves to holding Stats relative to their containing directory.

The performance difference is negligible, but the memory usage of the scandir node for ./pants dependencies :: in pantsbuild/pants drops by 32% (562754 -> 382163), and deeper / wider directory structures should notice a larger difference.

@stuhood stuhood force-pushed the stuhood/relative-scandir branch from 3a3c57a to 3c580bc Compare June 4, 2023 19:37
@stuhood stuhood changed the title scandir returns relative Stats. scandir returns Stats relative to its directory. Jun 4, 2023
@stuhood stuhood requested review from huonw, tdyas and thejcannon June 4, 2023 20:58
@stuhood stuhood marked this pull request as ready for review June 4, 2023 20:58
@stuhood stuhood enabled auto-merge (squash) June 5, 2023 18:33
@stuhood stuhood merged commit db68d19 into main Jun 5, 2023
@stuhood stuhood deleted the stuhood/relative-scandir branch June 5, 2023 19:29
huonw added a commit that referenced this pull request Jun 14, 2023
This updates `changelog.py` to write the (non-internal) changes to the
relevant release notes file directly, rather than requiring a human to
copy-paste it in. For now, the file is just mutated, without committing.
The internal changes are still printed for the human to deal with.

For instance, `pants run src/python/pants_release/changelog.py --
--prior 2.18.0.dev1 --new 2.18.0.dev2` adds a new section to
`src/python/pants/notes/2.18.x.md`:

```diff
diff --git a/src/python/pants/notes/2.18.x.md b/src/python/pants/notes/2.18.x.md
index e648a45..d6668a24b1 100644
--- a/src/python/pants/notes/2.18.x.md
+++ b/src/python/pants/notes/2.18.x.md
@@ -1,5 +1,35 @@
 # 2.18.x Release Series
 
+## 2.18.0.dev2 (Jun 14, 2023)
+
+### New Features
+
+* Include complete platforms for FaaS environments for more reliable building ([#19253](#19253))
+
+* Add experimental support for Rustfmt ([#18842](#18842))
+
+* Helm deployment chart field ([#19234](#19234))
+
+### Plugin API Changes
+
+* Replace `include_special_cased_deps` flag with `should_traverse_deps_predicate` ([#19272](#19272))
+
+### Bug Fixes
+
+* Raise an error if isort can't read a config file ([#19294](#19294))
+
+* Improve handling of additional files in Helm unit tests ([#19263](#19263))
+
+* Add taplo to the release ([#19258](#19258))
+
+* Handle `from foo import *` wildcard imports in Rust dep inference parser ([#19249](#19249))
+
+* Support usage of `scala_artifact` addresses in `scalac_plugin` targets ([#19205](#19205))
+
+### Performance
+
+* `scandir` returns `Stat`s relative to its directory. ([#19246](#19246))
+
 ## 2.18.0.dev1 (Jun 02, 2023)
 
 ### New Features
```

This also moves it into the new `pants_release` root, adds some basic
tests, and has it fetch the relevant branch directly, rather than
prompting the user to do so. It also pulls out a `git.py` support module
with helpers for exec-ing `git` as an external process.

The commits are individually reviewable.

This is a step towards automating more of the "start release" process,
per #19279. After this
core refactoring of the functionality, I think the next step is to
combine it with the CONTRIBUTORS update and version bumping, and then
after that string it all together on CI so that starting a release is
fully automated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants