Make metadata a workproduct and reuse it #114669

cjgillot · 2023-08-09T19:57:37Z

This PR aims to skip the generation of metadata by reusing the infrastructure that already exists for compiled codegen-units, namely "workproducts".

This can yield substantial gains (~10%) when we can demonstrate that metadata does not change between an incremental session and the next. This is the case if the crate is unchanged, or if all the changes are in upstream crates and have no effect on it. This latter case is most interesting, as it arises regularly for users with several crates in their workspace.

TODO:

Materialize the fact that metadata encoding relies on the relative order of definitions;
Refactor the handling of doc links.

cjgillot · 2023-08-09T20:01:11Z

@bors try @rust-timer queue

bors · 2023-08-09T20:01:21Z

⌛ Trying commit 3895d701e4a4c854cc003e599ac6efd49435aaef with merge 6a3ac25372eb1f930be018a4061b9b281bb3f42f...

bors · 2023-08-09T22:11:00Z

☀️ Try build successful - checks-actions
Build commit: 6a3ac25372eb1f930be018a4061b9b281bb3f42f (6a3ac25372eb1f930be018a4061b9b281bb3f42f)

rust-timer · 2023-08-09T23:14:48Z

Finished benchmarking commit (6a3ac25372eb1f930be018a4061b9b281bb3f42f): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Warning ⚠: The following benchmark(s) failed to build:

hyper-0.14.18
exa-0.10.1
cranelift-codegen-0.82.1
bitmaps-3.1.0
stm32f4-0.14.0
ctfe-stress-5
helloworld
unused-warnings
cargo-0.60.0
image-0.24.1
libc-0.2.124
tt-muncher
issue-88862
issue-46449
ucd
regression-31157
unify-linearly
webrender-2022
deep-vector
unicode-normalization-0.1.19
await-call-tree
clap-3.1.6
many-assoc-items
serde-1.0.136
wg-grammar
html5ever-0.26.0
externs
issue-58319
diesel-1.4.8
regex-1.5.5
ripgrep-13.0.0
wf-projection-stress-65510
match-stress
token-stream-stress
serde_derive-1.0.136
coercions
projection-caching
deeply-nested-multi
syn-1.0.89
tuple-stress
derive

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.1%	[1.3%, 3.2%]	7
Regressions ❌ (secondary)	1.1%	[0.9%, 1.7%]	3
Improvements ✅ (primary)	-4.6%	[-17.9%, -2.4%]	30
Improvements ✅ (secondary)	-5.5%	[-10.8%, -1.9%]	19
All ❌✅ (primary)	-3.4%	[-17.9%, 3.2%]	37

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.8%, 2.3%]	11
Regressions ❌ (secondary)	2.1%	[1.3%, 2.9%]	2
Improvements ✅ (primary)	-1.6%	[-2.1%, -1.0%]	8
Improvements ✅ (secondary)	-2.8%	[-4.1%, -1.4%]	5
All ❌✅ (primary)	0.1%	[-2.1%, 2.3%]	19

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.6%	[1.6%, 3.3%]	10
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	1
Improvements ✅ (primary)	-7.2%	[-18.2%, -2.1%]	32
Improvements ✅ (secondary)	-8.1%	[-14.7%, -2.1%]	21
All ❌✅ (primary)	-4.9%	[-18.2%, 3.3%]	42

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-2.3%, -0.0%]	60
Improvements ✅ (secondary)	-0.8%	[-2.2%, -0.1%]	6
All ❌✅ (primary)	-0.3%	[-2.3%, -0.0%]	60

Bootstrap: 632.289s -> 632.478s (0.03%)

cjgillot · 2023-08-10T11:18:07Z

compiler/rustc_metadata/src/rmeta/encoder.rs

-        }
+        //for (def_id, traits) in &tcx.resolutions(()).doc_link_traits_in_scope {
+        //    record_array!(self.tables.doc_link_traits_in_scope[def_id.to_def_id()] <- traits);
+        //}


@petrochenkov I've been trying to avoid the dependency on resolutions by using doc_link_resolutions and doc_link_traits_in_scope queries. But I could not understand in which case a definition would have an entry in those, and in which case they would ICE.

Is there a general rule?
Query description says "for a module". In which sense? resolve or mod items?

Query description says "for a module". In which sense? resolve or mod items?

mod items (or crate root).
There will be an entry if a module has doc links on items inside it (only exported items or all items, depending on options, resolve/late.rs determines the precise logic).
This is the data that rustdoc currently wants from us.

The query ICEs instead of producing None because rustdoc is unreliable, and will have future bugs and unpredictable changes with high probability, this ICE prevents it from going out of sync with the logic in rustc.
Maybe you could use a "private" None-returning query used for metadata, and a public ICEing wrapper around it used by rustdoc.

cjgillot · 2023-08-12T08:24:28Z

@bors try @rust-timer queue

bors · 2023-08-12T08:24:36Z

⌛ Trying commit 5f13bb1 with merge 2fce67ca52380d54b739a68cac1efc7c468cbaa6...

bors · 2023-08-12T10:35:13Z

☀️ Try build successful - checks-actions
Build commit: 2fce67ca52380d54b739a68cac1efc7c468cbaa6 (2fce67ca52380d54b739a68cac1efc7c468cbaa6)

rust-timer · 2023-08-12T12:52:07Z

Finished benchmarking commit (2fce67ca52380d54b739a68cac1efc7c468cbaa6): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.6%, 3.2%]	52
Regressions ❌ (secondary)	1.0%	[0.5%, 1.8%]	12
Improvements ✅ (primary)	-4.6%	[-17.9%, -1.0%]	48
Improvements ✅ (secondary)	-5.0%	[-10.8%, -0.5%]	33
All ❌✅ (primary)	-1.5%	[-17.9%, 3.2%]	100

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.4%	[0.6%, 2.1%]	14
Regressions ❌ (secondary)	1.7%	[1.2%, 2.8%]	3
Improvements ✅ (primary)	-1.8%	[-3.9%, -0.7%]	20
Improvements ✅ (secondary)	-2.1%	[-4.1%, -1.0%]	8
All ❌✅ (primary)	-0.5%	[-3.9%, 2.1%]	34

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.4%	[0.7%, 3.9%]	20
Regressions ❌ (secondary)	3.2%	[3.2%, 3.2%]	1
Improvements ✅ (primary)	-7.0%	[-18.3%, -2.1%]	48
Improvements ✅ (secondary)	-8.3%	[-13.8%, -2.5%]	29
All ❌✅ (primary)	-4.2%	[-18.3%, 3.9%]	68

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.0%, 2.6%]	72
Regressions ❌ (secondary)	1.7%	[0.0%, 3.8%]	13
Improvements ✅ (primary)	-0.3%	[-0.9%, -0.0%]	21
Improvements ✅ (secondary)	-0.9%	[-2.2%, -0.1%]	10
All ❌✅ (primary)	0.3%	[-0.9%, 2.6%]	93

Bootstrap: 632.009s -> 632.421s (0.07%)

compiler/rustc_codegen_ssa/src/back/write.rs

bors · 2023-10-01T14:34:17Z

⌛ Trying commit 9742c49 with merge 9310a5e...

Make metadata a workproduct and reuse it This PR aims to skip the generation of metadata by reusing the infrastructure that already exists for compiled codegen-units, namely "workproducts". This can yield substantial gains (~10%) when we can demonstrate that metadata does not change between an incremental session and the next. This is the case if the crate is unchanged, or if all the changes are in upstream crates and have no effect on it. This latter case is most interesting, as it arises regularly for users with several crates in their workspace. TODO: - [ ] Materialize the fact that metadata encoding relies on the relative order of definitions; - [ ] Refactor the handling of doc links.

bors · 2023-10-01T15:44:15Z

☀️ Try build successful - checks-actions
Build commit: 9310a5e (9310a5eb5f8c957370c182edcd92216721062711)

rust-timer · 2023-10-01T17:20:33Z

Finished benchmarking commit (9310a5e): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.2%, 2.9%]	68
Regressions ❌ (secondary)	0.9%	[0.4%, 1.7%]	16
Improvements ✅ (primary)	-3.9%	[-9.0%, -1.2%]	48
Improvements ✅ (secondary)	-4.7%	[-10.9%, -0.5%]	33
All ❌✅ (primary)	-1.0%	[-9.0%, 2.9%]	116

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[0.7%, 3.6%]	19
Regressions ❌ (secondary)	1.7%	[1.5%, 2.2%]	3
Improvements ✅ (primary)	-1.8%	[-2.9%, -1.0%]	12
Improvements ✅ (secondary)	-2.2%	[-2.4%, -1.9%]	3
All ❌✅ (primary)	0.5%	[-2.9%, 3.6%]	31

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.1%	[1.0%, 3.7%]	40
Regressions ❌ (secondary)	2.8%	[2.4%, 3.2%]	3
Improvements ✅ (primary)	-6.2%	[-12.5%, -1.8%]	48
Improvements ✅ (secondary)	-8.5%	[-13.8%, -3.1%]	28
All ❌✅ (primary)	-2.4%	[-12.5%, 3.7%]	88

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.0%, 2.0%]	67
Regressions ❌ (secondary)	1.3%	[0.0%, 3.0%]	13
Improvements ✅ (primary)	-0.2%	[-0.6%, -0.0%]	21
Improvements ✅ (secondary)	-0.8%	[-2.2%, -0.1%]	10
All ❌✅ (primary)	0.2%	[-0.6%, 2.0%]	88

Bootstrap: 630.664s -> 631.304s (0.10%)
Artifact size: 317.36 MiB -> 317.40 MiB (0.01%)

bjorn3 · 2023-10-01T19:22:14Z

compiler/rustc_codegen_cranelift/src/driver/aot.rs

@@ -56,6 +56,21 @@ impl OngoingCodegen {
        backend_config: &BackendConfig,
    ) -> (CodegenResults, FxIndexMap<WorkProductId, WorkProduct>) {
        let mut work_products = FxIndexMap::default();
+
+        if !backend_config.disable_incr_cache {


Reusing the crate metadata when disable_incr_cache is set is fine. This option exists to allow using incr comp while still recreating all object files. This allows for faster iteration of cg_clif changes by allowing the frontend results to be cached while still respecting changes to cg_clif.

bjorn3 · 2023-10-01T19:25:46Z

compiler/rustc_codegen_ssa/src/back/write.rs

+        {
+            work_products.insert(id, product);
+        }
+    }


This should be done for all codegen backends. Maybe move it to rustc_incremental or rustc_interface? rustc_interface is responsible for ensuring that the .rmeta file is written in the first place.

rustc_interface has the .rmeta file written by calling encode_and_write_metadata(tcx). This specific code path fetches the metadata to put into the rlib file.

rustbot · 2024-08-24T16:07:29Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

alex-semenyuk · 2024-09-21T14:20:48Z

From triage. @cjgillot Am I right that comments were addressed and it's ready for review?

petrochenkov · 2024-09-22T11:27:17Z

From triage. @cjgillot Am I right that comments were addressed and it's ready for review?

Waiting on cjgillot to answer.
@rustbot author

bors · 2024-09-23T04:36:47Z

☔ The latest upstream changes (presumably #130724) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-01-17T00:22:15Z

☔ The latest upstream changes (presumably #135592) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2025-02-19T15:38:25Z

Status is not very clear, still waiting on author.
@cjgillot Do you plan to work on completing it this year?

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 9, 2023

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 9, 2023

cjgillot commented Aug 10, 2023

View reviewed changes

cjgillot force-pushed the metadata-wp branch from 3895d70 to 5f13bb1 Compare August 12, 2023 08:13

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 12, 2023

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 12, 2023

petrochenkov self-assigned this Aug 13, 2023

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 13, 2023

bjorn3 reviewed Aug 16, 2023

View reviewed changes

compiler/rustc_codegen_ssa/src/back/write.rs Show resolved Hide resolved

cjgillot force-pushed the metadata-wp branch from 5f13bb1 to 9742c49 Compare October 1, 2023 14:33

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 1, 2023

bjorn3 reviewed Oct 1, 2023

View reviewed changes

cjgillot force-pushed the metadata-wp branch from 9742c49 to 031c9fd Compare August 3, 2024 16:41

cjgillot force-pushed the metadata-wp branch from 031c9fd to ebf19c5 Compare August 24, 2024 15:59

cjgillot marked this pull request as ready for review August 24, 2024 16:07

alex-semenyuk added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 21, 2024

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 22, 2024

cjgillot force-pushed the metadata-wp branch from ebf19c5 to 3eb1b32 Compare October 5, 2024 18:13

cjgillot added 6 commits October 30, 2024 16:22

Save metadata among work products.

1533298

Reuse metadata file from work products.

b8d9359

Querify module_children_local.

cdbeccc

Ensure doc link maps have one entry per module.

1b41853

Do not depend on FOREVER_RED_NODE.

0f1f693

Port metadata reuse to cg_clif.

4f8f35d

cjgillot force-pushed the metadata-wp branch from 3eb1b32 to 4f8f35d Compare October 30, 2024 16:56

Do not gate clif of !backend_config.disable_incr_cache.

214e49e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make metadata a workproduct and reuse it #114669

Make metadata a workproduct and reuse it #114669

cjgillot commented Aug 9, 2023 •

edited

Loading

cjgillot commented Aug 9, 2023

This comment has been minimized.

bors commented Aug 9, 2023

This comment has been minimized.

bors commented Aug 9, 2023

This comment has been minimized.

rust-timer commented Aug 9, 2023

cjgillot Aug 10, 2023

petrochenkov Aug 13, 2023

cjgillot commented Aug 12, 2023

This comment has been minimized.

bors commented Aug 12, 2023

bors commented Aug 12, 2023

This comment has been minimized.

rust-timer commented Aug 12, 2023

bors commented Oct 1, 2023

bors commented Oct 1, 2023

This comment has been minimized.

rust-timer commented Oct 1, 2023

bjorn3 Oct 1, 2023

bjorn3 Oct 1, 2023

cjgillot Oct 30, 2024

rustbot commented Aug 24, 2024

alex-semenyuk commented Sep 21, 2024 •

edited

Loading

petrochenkov commented Sep 22, 2024

bors commented Sep 23, 2024

bors commented Jan 17, 2025

petrochenkov commented Feb 19, 2025

Make metadata a workproduct and reuse it #114669

Are you sure you want to change the base?

Make metadata a workproduct and reuse it #114669

Conversation

cjgillot commented Aug 9, 2023 • edited Loading

cjgillot commented Aug 9, 2023

This comment has been minimized.

bors commented Aug 9, 2023

This comment has been minimized.

bors commented Aug 9, 2023

This comment has been minimized.

rust-timer commented Aug 9, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjgillot commented Aug 12, 2023

This comment has been minimized.

bors commented Aug 12, 2023

bors commented Aug 12, 2023

This comment has been minimized.

rust-timer commented Aug 12, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

bors commented Oct 1, 2023

bors commented Oct 1, 2023

This comment has been minimized.

rust-timer commented Oct 1, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rustbot commented Aug 24, 2024

alex-semenyuk commented Sep 21, 2024 • edited Loading

petrochenkov commented Sep 22, 2024

bors commented Sep 23, 2024

bors commented Jan 17, 2025

petrochenkov commented Feb 19, 2025

cjgillot commented Aug 9, 2023 •

edited

Loading

alex-semenyuk commented Sep 21, 2024 •

edited

Loading