Skip to content

[ty] Compact retained definition and expression identities#25606

Merged
charliermarsh merged 2 commits into
mainfrom
charlie/compact-definition-scope
Jun 4, 2026
Merged

[ty] Compact retained definition and expression identities#25606
charliermarsh merged 2 commits into
mainfrom
charlie/compact-definition-scope

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Jun 3, 2026

Copy link
Copy Markdown
Member

Summary

The semantic index includes a huge number of Definition and Expression ingredients. Both retain file and file-local scope fields, even though the existing ScopeId identifies both.

This PR modifies the representations to store definition and expression scopes through ScopeId, and pack a definition place and re-export state into one enum field. This preserves the existing accessors while reducing the fields retained by each tracked ingredient.

I don't think I'm breaking any Salsa rules here, but I've been wrong before!

@astral-sh-bot astral-sh-bot Bot added the ty Multi-file analysis & type inference label Jun 3, 2026
@astral-sh-bot

astral-sh-bot Bot commented Jun 3, 2026

Copy link
Copy Markdown

Typing conformance results

No changes detected ✅

Current numbers
The percentage of diagnostics emitted that were expected errors held steady at 92.13%. The percentage of expected errors that received a diagnostic held steady at 87.18%. The number of fully passing files held steady at 92/134.

@astral-sh-bot

astral-sh-bot Bot commented Jun 3, 2026

Copy link
Copy Markdown

Memory usage report

Summary

Project Old New Diff Outcome
flake8 39.61MB 39.38MB -0.57% (229.87kB) ⬇️
trio 101.32MB 100.78MB -0.53% (551.20kB) ⬇️
sphinx 245.12MB 243.95MB -0.48% (1.17MB) ⬇️
prefect 670.24MB 666.88MB -0.50% (3.36MB) ⬇️

Significant changes

Click to expand detailed breakdown

flake8

Name Old New Diff Outcome
Definition 2.00MB 1.82MB -9.09% (185.90kB) ⬇️
Expression 361.34kB 321.19kB -11.11% (40.15kB) ⬇️
all_narrowing_constraints_for_expression 108.41kB 106.58kB -1.69% (1.83kB) ⬇️
all_negative_narrowing_constraints_for_expression 105.14kB 103.33kB -1.73% (1.82kB) ⬇️
loop_header_reachability 13.09kB 12.93kB -1.25% (168.00B) ⬇️
place_by_id 141.13kB 141.12kB -0.01% (12.00B) ⬇️

trio

Name Old New Diff Outcome
Definition 3.97MB 3.61MB -9.09% (369.04kB) ⬇️
Expression 1.42MB 1.26MB -11.11% (161.53kB) ⬇️
all_narrowing_constraints_for_expression 714.16kB 703.90kB -1.44% (10.27kB) ⬇️
all_negative_narrowing_constraints_for_expression 681.56kB 671.79kB -1.43% (9.77kB) ⬇️
loop_header_reachability 138.54kB 137.97kB -0.41% (588.00B) ⬇️
place_by_id 558.23kB 558.22kB -0.00% (12.00B) ⬇️

sphinx

Name Old New Diff Outcome
Definition 8.15MB 7.40MB -9.09% (758.20kB) ⬇️
Expression 3.22MB 2.86MB -11.11% (366.34kB) ⬇️
all_narrowing_constraints_for_expression 2.75MB 2.72MB -1.27% (35.77kB) ⬇️
all_negative_narrowing_constraints_for_expression 2.68MB 2.65MB -1.26% (34.64kB) ⬇️
loop_header_reachability 367.35kB 364.31kB -0.83% (3.04kB) ⬇️
place_by_id 1.37MB 1.37MB -0.00% (48.00B) ⬇️

prefect

Name Old New Diff Outcome
Definition 22.22MB 20.21MB -9.08% (2.02MB) ⬇️
Expression 9.72MB 8.64MB -11.11% (1.08MB) ⬇️
all_narrowing_constraints_for_expression 8.55MB 8.42MB -1.50% (131.16kB) ⬇️
all_negative_narrowing_constraints_for_expression 8.38MB 8.25MB -1.52% (130.76kB) ⬇️
loop_header_reachability 479.40kB 475.72kB -0.77% (3.68kB) ⬇️
place_by_id 5.01MB 5.01MB -0.00% (60.00B) ⬇️

@astral-sh-bot

astral-sh-bot Bot commented Jun 3, 2026

Copy link
Copy Markdown

ecosystem-analyzer results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

@charliermarsh charliermarsh force-pushed the charlie/compact-definition-scope branch from 39d0b86 to 2f4082b Compare June 3, 2026 18:55
@charliermarsh charliermarsh changed the title [ty] Compact retained semantic identities [ty] Compact retained definition and expression identities Jun 3, 2026
@charliermarsh charliermarsh force-pushed the charlie/compact-definition-scope branch from 2f4082b to 0de40d5 Compare June 3, 2026 20:03
@charliermarsh charliermarsh added the performance Potential performance improvement label Jun 3, 2026
@charliermarsh charliermarsh marked this pull request as ready for review June 3, 2026 20:05
@astral-sh-bot astral-sh-bot Bot requested a review from MichaReiser June 3, 2026 20:05
Comment thread crates/ty_python_core/src/builder.rs Outdated
self.db,
self.file,
self.current_scope(),
self.scope_ids_by_scope[self.current_scope()],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe add a self.current_scoipe_id() helper

Comment thread crates/ty_python_core/src/definition.rs Outdated
#[derive(
Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, salsa::Update, get_size2::GetSize,
)]
#[doc(hidden)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you say more why you decided to hide this type?

Comment thread crates/ty_python_core/src/definition.rs Outdated
}

#[derive(
Debug, Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, salsa::Update, get_size2::GetSize,

@MichaReiser MichaReiser Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need all the derives here? E.g. PartialOrd? Hash? I don't really see a natural ordering for those.

},
Member {
id: ScopedMemberId,
is_reexported: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth adding a comment saying why is_reexported is here instead of Definition. Or overall, adding a comment why this type exists in the first place. I understand is that DefinitionPlace has a nieche whereas having the bool on Definition adds 7 byte padding


/// The scope in which the expression occurs.
pub file_scope: FileScopeId,
pub scope_id: ScopeId<'db>,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trade off here is reducing memory usage (by using the interned ScopeId) at the cost of a few more reads that must go through db (looking up the FileScopeId and File both need to go through db)

Comment thread crates/ty_python_core/src/expression.rs Outdated
Comment on lines 71 to 73
pub fn file_scope(self, db: &'db dyn Db) -> FileScopeId {
self.scope_id(db).file_scope_id(db)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method seems unused

Comment thread crates/ty_python_core/src/definition.rs Outdated
impl get_size2::GetSize for Definition<'_> {}

impl<'db> Definition<'db> {
pub(crate) fn create(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use new here and change the Salsa generated constructor by using

constructor = new_internal,

@MichaReiser MichaReiser added memory and removed performance Potential performance improvement labels Jun 4, 2026
@charliermarsh charliermarsh force-pushed the charlie/compact-definition-scope branch from 0de40d5 to 9d2f528 Compare June 4, 2026 10:56
@charliermarsh

Copy link
Copy Markdown
Member Author

Thank you ❤️

@charliermarsh charliermarsh enabled auto-merge (squash) June 4, 2026 10:58
@charliermarsh charliermarsh merged commit 27058fc into main Jun 4, 2026
58 checks passed
@charliermarsh charliermarsh deleted the charlie/compact-definition-scope branch June 4, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

memory ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants