Split the node_id_to_def_id table into a per-owner table by oli-obk · Pull Request #138995 · rust-lang/rust

oli-obk · 2025-03-26T18:23:03Z

My goal is to split all the resolver tables that get passed to act lowering into per-owner tables, so that all information that ast lowering needs from the resolver is separated by owners. This should allow us to fully split ast lowering to have one query invocation per owner that steal the individual resolver results for each owner.

part of rust-lang/rust-project-goals#620

oli-obk · 2025-03-26T18:23:14Z

@bors try @rust-timer queue

bors · 2025-03-26T18:24:25Z

⌛ Trying commit 33f5615 with merge 792af13...

[perf experiment] Split the resolver tables into per-owner tables r? `@ghost` just doing some experiments to see if splitting `hir_crate` is feasible by checking if splitting the resolver's output into per-owner queries is feasible (rust-lang#95004) Basically necessary for rust-lang#138705 as that can't be landed perf-wise while the `hir_crate` query is still a thing

bors · 2025-03-26T20:30:56Z

☀️ Try build successful - checks-actions
Build commit: 792af13 (792af13061770b940e351039beebe10bd97d4627)

rust-timer · 2025-03-26T21:48:18Z

Finished benchmarking commit (792af13): comparison URL.

Overall result: ❌ regressions - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.1%, 0.6%]	129
Regressions ❌ (secondary)	0.5%	[0.1%, 1.5%]	69
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.1%, 0.6%]	129

Max RSS (memory usage)

Results (primary 1.3%, secondary 0.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.3%	[0.9%, 1.7%]	5
Regressions ❌ (secondary)	1.8%	[1.0%, 2.8%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.0%	[-2.4%, -1.6%]	2
All ❌✅ (primary)	1.3%	[0.9%, 1.7%]	5

Cycles

Results (secondary 1.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.2%	[1.5%, 2.6%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.0%	[-2.7%, -1.2%]	2
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 777.548s -> 776.554s (-0.13%)
Artifact size: 365.81 MiB -> 365.80 MiB (-0.01%)

oli-obk · 2025-03-28T12:20:32Z

@bors try @rust-timer queue

bors · 2025-03-28T12:21:45Z

⌛ Trying commit a16a6f1 with merge 66f172c...

[perf experiment] Split the resolver tables into per-owner tables r? `@ghost` just doing some experiments to see if splitting `hir_crate` is feasible by checking if splitting the resolver's output into per-owner queries is feasible (rust-lang#95004) Basically necessary for rust-lang#138705 as that can't be landed perf-wise while the `hir_crate` query is still a thing

bors · 2025-03-28T14:29:54Z

☀️ Try build successful - checks-actions
Build commit: 66f172c (66f172c845b537c43e7e41f92eaf99957253f6bd)

rust-timer · 2025-03-28T16:04:26Z

Finished benchmarking commit (66f172c): comparison URL.

Overall result: ❌ regressions - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.1%, 0.8%]	29
Regressions ❌ (secondary)	0.8%	[0.3%, 2.1%]	30
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.2%, -0.2%]	2
All ❌✅ (primary)	0.3%	[0.1%, 0.8%]	29

Max RSS (memory usage)

Results (primary 1.6%, secondary 2.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.6%	[1.0%, 2.1%]	5
Regressions ❌ (secondary)	3.9%	[2.0%, 7.4%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.2%	[-1.3%, -1.1%]	2
All ❌✅ (primary)	1.6%	[1.0%, 2.1%]	5

Cycles

Results (secondary 2.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.9%	[2.0%, 3.6%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 778.99s -> 777.791s (-0.15%)
Artifact size: 365.92 MiB -> 365.97 MiB (0.01%)

oli-obk · 2025-03-28T16:08:05Z

ok... better, but not great yet either

oli-obk · 2025-03-31T16:06:25Z

@bors try @rust-timer queue

[perf experiment] Split the resolver tables into per-owner tables r? `@ghost` just doing some experiments to see if splitting `hir_crate` is feasible by checking if splitting the resolver's output into per-owner queries is feasible (rust-lang#95004) Basically necessary for rust-lang#138705 as that can't be landed perf-wise while the `hir_crate` query is still a thing

bors · 2025-03-31T16:07:37Z

⌛ Trying commit a78e1a6 with merge 012f3ee...

bors · 2025-03-31T18:15:53Z

☀️ Try build successful - checks-actions
Build commit: 012f3ee (012f3eec5acc351e2cb934444de98793bf94c9e7)

oli-obk · 2026-05-08T06:32:55Z

@bors try @rust-timer queue

[perf experiment] Split the resolver tables into per-owner tables

rust-bors · 2026-05-08T08:42:38Z

☀️ Try build successful (CI)
Build commit: b4dc833 (b4dc83368f4dfd4c90eb2dc5ed7fce8af6bda003, parent: 63b1dfc0e00fd6f8ad7cd8817fc712e7d9b7be59)

rust-timer · 2026-05-08T09:33:00Z

Finished benchmarking commit (b4dc833): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.1%, 0.2%]	5
Regressions ❌ (secondary)	0.5%	[0.1%, 0.8%]	13
Improvements ✅ (primary)	-0.4%	[-0.6%, -0.3%]	5
Improvements ✅ (secondary)	-0.3%	[-0.4%, -0.1%]	5
All ❌✅ (primary)	-0.1%	[-0.6%, 0.2%]	10

Max RSS (memory usage)

Results (primary 0.3%, secondary 1.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[1.8%, 2.6%]	4
Regressions ❌ (secondary)	3.9%	[2.1%, 5.7%]	15
Improvements ✅ (primary)	-1.7%	[-2.1%, -1.4%]	4
Improvements ✅ (secondary)	-2.9%	[-4.3%, -0.7%]	8
All ❌✅ (primary)	0.3%	[-2.1%, 2.6%]	8

Cycles

Results (primary -4.2%, secondary -5.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.0%	[2.3%, 4.1%]	6
Improvements ✅ (primary)	-4.2%	[-5.5%, -2.9%]	2
Improvements ✅ (secondary)	-21.3%	[-23.8%, -19.7%]	3
All ❌✅ (primary)	-4.2%	[-5.5%, -2.9%]	2

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 498.27s -> 499.245s (0.20%)
Artifact size: 395.08 MiB -> 395.09 MiB (0.00%)

oli-obk · 2026-05-08T12:30:46Z

yay, getting there

oli-obk · 2026-05-08T12:52:47Z

unused-warnings is a small regression, but somewhat expected considering the high number of unused imports that all run some extra logic for obtaining information from the hash tables.

the primary regressions (serde, bitmaps, clap) are very small, largely inliner noise (local testing says so) and to a smaller part because of some extra malloc invocations for the extra indirection of the hash maps.

Overall it's a wash, so I think we should just merge it. Especially considering that some of the refactorings that enabled this being perf neutral were perf improvements themselves.

@rustbot ready

rustbot · 2026-05-08T12:52:53Z

The Clippy subtree was changed

cc @rust-lang/clippy

…id_to_def_id`

Now they unfortunately land in their parent, which is not necessary

rustbot · 2026-05-08T16:00:07Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

petrochenkov · 2026-05-08T14:39:09Z

+        F: Fn(T) -> UnordItems<O, U>,
    {
-        UnordItems(self.0.flat_map(f))
+        UnordItems(self.0.flat_map(move |x| f(x).0))


I don't understand the motivation behind these changes, could you explain?

View changes since the review

The follow-up commit uses flat_map in def_id_to_node_id. To flatten NodeMap<PerOwner...>, we need an UnordItems of the owners table to be able to be flat_map its inner node_id_to_def_id, which is also unord

petrochenkov · 2026-05-08T14:41:03Z


 impl<T, I: Iterator<Item = T>> UnordItems<T, I> {
+    #[inline]
+    pub fn wrap(iter: I) -> Self {


Suggested change

pub fn wrap(iter: I) -> Self {

pub fn new(iter: I) -> Self {

More idiomatic naming for the constructor.

View changes since the review

petrochenkov · 2026-05-08T14:43:09Z

 impl<T, I: Iterator<Item = T>> UnordItems<T, I> {
+    #[inline]
+    pub fn wrap(iter: I) -> Self {
+        Self(iter)


Suggested change

Self(iter)

UnordItems(iter)

Could you avoid Self in non-generic code, at least in constructors like Self { ... } or Self(...)? Makes it hard to search code.
There are several instances in this PR.

View changes since the review

petrochenkov · 2026-05-08T14:46:45Z

    /// The id of the owner
    pub id: ast::NodeId,
+    /// The `DefId` of the owner, can't be found in `node_id_to_def_id`.
+    pub def_id: LocalDefId,


Is this an optimization?
Or putting it into node_id_to_def_id was a hack in the first place?

View changes since the review

Putting it in the hash map was a "preserve more previous behaviour" thing. Just wanted to keep the diff small. I have a commit somewhere from last year where i added the DefId here directly. In fact, it should be an OwnerId, and then we can get rid of the one from the ast lowerer.

But yes, for the purposes of this PR it's an optimization.

petrochenkov · 2026-05-08T16:21:11Z

(Will continue reviewing tomorrow or later today.)

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 26, 2025

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 26, 2025

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Mar 26, 2025

oli-obk force-pushed the split-resolver branch from 33f5615 to a16a6f1 Compare March 28, 2025 12:17

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 28, 2025

bors mentioned this pull request Mar 28, 2025

Fix TAIT & ATPIT feature gating in the presence of anon consts #139063

Merged

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 28, 2025

oli-obk force-pushed the split-resolver branch from a16a6f1 to a78e1a6 Compare March 31, 2025 16:06

This comment has been minimized.

Sign in to view

oli-obk force-pushed the split-resolver branch from 9d57833 to a15fcac Compare May 8, 2026 06:32

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request May 8, 2026

Auto merge of #138995 - oli-obk:split-resolver, r=<try>

b4dc833

[perf experiment] Split the resolver tables into per-owner tables

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 8, 2026

This comment has been minimized.

Sign in to view

oli-obk force-pushed the split-resolver branch from a15fcac to 1229820 Compare May 8, 2026 08:59

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 8, 2026

oli-obk changed the title ~~[perf experiment] Split the resolver tables into per-owner tables~~ Split the resolver tables into per-owner tables May 8, 2026

oli-obk changed the title ~~Split the resolver tables into per-owner tables~~ Split the node_id_to_def_id table into a per-owner table May 8, 2026

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label May 8, 2026

oli-obk marked this pull request as ready for review May 8, 2026 12:52

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 8, 2026

This comment has been minimized.

Sign in to view

oli-obk and others added 5 commits May 8, 2026 17:32

Split trait item resolving into its own method

9b80a68

Make UnordItems::flat_map take UnordItems as args, too

f39d529

Split the resolver tables into per-owner tables, starting with `node_…

041a571

…id_to_def_id`

Iterate over all modules to find cfg-ed out items via their NodeId

3a906e0

Stop putting owner's DefId into their own node_id_to_def_id map

641d6fc

Now they unfortunately land in their parent, which is not necessary

oli-obk force-pushed the split-resolver branch from 1229820 to 641d6fc Compare May 8, 2026 16:00

petrochenkov reviewed May 8, 2026

View reviewed changes

Uh oh!

Conversation

oli-obk commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oli-obk commented Mar 26, 2025

Uh oh!

This comment has been minimized.

bors commented Mar 26, 2025

Uh oh!

bors commented Mar 26, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Mar 26, 2025

Overall result: ❌ regressions - please read the text below

Uh oh!

oli-obk commented Mar 28, 2025

Uh oh!

This comment has been minimized.

bors commented Mar 28, 2025

Uh oh!

bors commented Mar 28, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Mar 28, 2025

Overall result: ❌ regressions - please read the text below

Uh oh!

oli-obk commented Mar 28, 2025

Uh oh!

oli-obk commented Mar 31, 2025

Uh oh!

bors commented Mar 31, 2025

Uh oh!

This comment has been minimized.

bors commented Mar 31, 2025

Uh oh!

oli-obk commented May 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented May 8, 2026

Uh oh!

This comment has been minimized.

rust-timer commented May 8, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

oli-obk commented May 8, 2026

Uh oh!

oli-obk commented May 8, 2026

Uh oh!

rustbot commented May 8, 2026

Uh oh!

This comment has been minimized.

rustbot commented May 8, 2026

Uh oh!

petrochenkov May 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov May 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov May 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov May 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov commented May 8, 2026

Uh oh!

Reviewers

Assignees

oli-obk commented Mar 26, 2025 •

edited

Loading

petrochenkov May 8, 2026 •

edited by rustbot

Loading

petrochenkov May 8, 2026 •

edited by rustbot

Loading

petrochenkov May 8, 2026 •

edited by rustbot

Loading

petrochenkov May 8, 2026 •

edited by rustbot

Loading