Auto merge of #128534 - bjorn3:split_stdlib_workspace, r=Mark-Simulacrum

Move the standard library to a separate workspace

This ensures that the Cargo.lock packaged for it in the rust-src component is up-to-date, allowing rust-analyzer to run cargo metadata on the standard library even when the rust-src component is stored in a read-only location as is necessary for loading crates.io dependencies of the standard library.

This also simplifies tidy's license check for runtime dependencies as it can now look at all entries in library/Cargo.lock without having to filter for just the dependencies of runtime crates. In addition this allows removing an exception in check_runtime_license_exceptions that was necessary due to the compiler enabling a feature on the object crate which pulls in a dependency not allowed for the standard library.

While cargo workspaces normally enable dependencies of multiple targets to be reused, for the standard library we do not want this reusing to prevent conflicts between dependencies of the sysroot and of tools that are built using this sysroot. For this reason we already use an unstable cargo feature to ensure that any dependencies which would otherwise be shared get a different -Cmetadata argument as well as using separate build dirs.

This doesn't change the situation around vendoring. We already have several cargo workspaces that need to be vendored. Adding another one doesn't change much.

There are also no cargo profiles that are shared between the root workspace and the library workspace anyway, so it doesn't add any extra work when changing cargo profiles.

Author: bors

.gitignore

@@ -48,6 +48,7 @@ build/
 /dist/
 /unicode-downloads
 /target
+/library/target
 /src/bootstrap/target
 /src/tools/x/target
 # Created by default with `src/ci/docker/run.sh`

Cargo.lock

@@ -2,8 +2,6 @@
 resolver = "1"
 members = [
   "compiler/rustc",
-  "library/std",
-  "library/sysroot",
   "src/etc/test-float-parse",
   "src/rustdoc-json-types",
   "src/tools/build_helper",
@@ -61,43 +59,15 @@ exclude = [
   # not all `Cargo.toml` files are available, so we exclude the `x` binary,
   # so it can be invoked before the current checkout is set up.
   "src/tools/x",
-  # stdarch has its own Cargo workspace
-  "library/stdarch",
 ]
 
-[profile.release.package.compiler_builtins]
-# For compiler-builtins we always use a high number of codegen units.
-# The goal here is to place every single intrinsic into its own object
-# file to avoid symbol clashes with the system libgcc if possible. Note
-# that this number doesn't actually produce this many object files, we
-# just don't create more than this number of object files.
-#
-# It's a bit of a bummer that we have to pass this here, unfortunately.
-# Ideally this would be specified through an env var to Cargo so Cargo
-# knows how many CGUs are for this specific crate, but for now
-# per-crate configuration isn't specifiable in the environment.
-codegen-units = 10000
-
 [profile.release.package.rustc-rayon-core]
 # The rustc fork of Rayon has deadlock detection code which intermittently
 # causes overflows in the CI (see https://github.com/rust-lang/rust/issues/90227)
 # so we turn overflow checks off for now.
 # FIXME: This workaround should be removed once #90227 is fixed.
 overflow-checks = false
 
-# These dependencies of the standard library implement symbolication for
-# backtraces on most platforms. Their debuginfo causes both linking to be slower
-# (more data to chew through) and binaries to be larger without really all that
-# much benefit. This section turns them all to down to have no debuginfo which
-# helps to improve link times a little bit.
-[profile.release.package]
-addr2line.debug = 0
-adler.debug = 0
-gimli.debug = 0
-miniz_oxide.debug = 0
-object.debug = 0
-rustc-demangle.debug = 0
-
 # These are very thin wrappers around executing lld with the right binary name.
 # Basically nothing within them can go wrong without having been explicitly logged anyway.
 # We ship these in every rustc tarball and even after compression they add up
@@ -120,10 +90,3 @@ codegen-units = 1
 # FIXME: LTO cannot be enabled for binaries in a workspace
 # <https://github.com/rust-lang/cargo/issues/9330>
 # lto = true
-
-[patch.crates-io]
-# See comments in `library/rustc-std-workspace-core/README.md` for what's going on
-# here
-rustc-std-workspace-core = { path = 'library/rustc-std-workspace-core' }
-rustc-std-workspace-alloc = { path = 'library/rustc-std-workspace-alloc' }
-rustc-std-workspace-std = { path = 'library/rustc-std-workspace-std' }

Cargo.toml

@@ -0,0 +1,44 @@
+[workspace]
+resolver = "1"
+members = [
+  "std",
+  "sysroot",
+]
+
+exclude = [
+  # stdarch has its own Cargo workspace
+  "stdarch",
+]
+
+[profile.release.package.compiler_builtins]
+# For compiler-builtins we always use a high number of codegen units.
+# The goal here is to place every single intrinsic into its own object
+# file to avoid symbol clashes with the system libgcc if possible. Note
+# that this number doesn't actually produce this many object files, we
+# just don't create more than this number of object files.
+#
+# It's a bit of a bummer that we have to pass this here, unfortunately.
+# Ideally this would be specified through an env var to Cargo so Cargo
+# knows how many CGUs are for this specific crate, but for now
+# per-crate configuration isn't specifiable in the environment.
+codegen-units = 10000
+
+# These dependencies of the standard library implement symbolication for
+# backtraces on most platforms. Their debuginfo causes both linking to be slower
+# (more data to chew through) and binaries to be larger without really all that
+# much benefit. This section turns them all to down to have no debuginfo which
+# helps to improve link times a little bit.
+[profile.release.package]
+addr2line.debug = 0
+adler.debug = 0
+gimli.debug = 0
+miniz_oxide.debug = 0
+object.debug = 0
+rustc-demangle.debug = 0
+
+[patch.crates-io]
+# See comments in `library/rustc-std-workspace-core/README.md` for what's going on
+# here
+rustc-std-workspace-core = { path = 'rustc-std-workspace-core' }
+rustc-std-workspace-alloc = { path = 'rustc-std-workspace-alloc' }
+rustc-std-workspace-std = { path = 'rustc-std-workspace-std' }

library/Cargo.lock

@@ -1034,6 +1034,8 @@ impl Step for PlainSourceTarball {
                 .arg("--sync")
                 .arg(builder.src.join("./compiler/rustc_codegen_gcc/Cargo.toml"))
                 .arg("--sync")
+                .arg(builder.src.join("./library/Cargo.toml"))
+                .arg("--sync")
                 .arg(builder.src.join("./src/bootstrap/Cargo.toml"))
                 .arg("--sync")
                 .arg(builder.src.join("./src/tools/opt-dist/Cargo.toml"))

library/Cargo.toml

@@ -47,6 +47,7 @@ impl Step for Vendor {
             "src/tools/rust-analyzer/Cargo.toml",
             "compiler/rustc_codegen_cranelift/Cargo.toml",
             "compiler/rustc_codegen_gcc/Cargo.toml",
+            "library/Cargo.toml",
             "src/bootstrap/Cargo.toml",
             "src/tools/rustbook/Cargo.toml",
         ] {

src/bootstrap/src/core/build_steps/dist.rs

@@ -86,6 +86,9 @@ fn workspace_members(build: &Build) -> Vec<Package> {
         packages
     };
 
-    // Collects `metadata.packages` from all workspaces.
-    collect_metadata("Cargo.toml")
+    // Collects `metadata.packages` from the root and library workspaces.
+    let mut packages = vec![];
+    packages.extend(collect_metadata("Cargo.toml"));
+    packages.extend(collect_metadata("library/Cargo.toml"));
+    packages
 }

src/bootstrap/src/core/build_steps/vendor.rs

@@ -54,6 +54,7 @@ type ExceptionList = &'static [(&'static str, &'static str)];
 pub(crate) const WORKSPACES: &[(&str, ExceptionList, Option<(&[&str], &[&str])>, &[&str])] = &[
     // The root workspace has to be first for check_rustfix to work.
     (".", EXCEPTIONS, Some((&["rustc-main"], PERMITTED_RUSTC_DEPENDENCIES)), &[]),
+    ("library", EXCEPTIONS_STDLIB, Some((&["sysroot"], PERMITTED_STDLIB_DEPENDENCIES)), &[]),
     // Outside of the alphabetical section because rustfmt formats it using multiple lines.
     (
         "compiler/rustc_codegen_cranelift",
@@ -90,7 +91,6 @@ const EXCEPTIONS: ExceptionList = &[
     ("colored", "MPL-2.0"),                                  // rustfmt
     ("dissimilar", "Apache-2.0"),                            // rustdoc, rustc_lexer (few tests) via expect-test, (dev deps)
     ("fluent-langneg", "Apache-2.0"),                        // rustc (fluent translations)
-    ("fortanix-sgx-abi", "MPL-2.0"),                         // libstd but only for `sgx` target. FIXME: this dependency violates the documentation comment above.
     ("instant", "BSD-3-Clause"),                             // rustc_driver/tracing-subscriber/parking_lot
     ("mdbook", "MPL-2.0"),                                   // mdbook
     ("option-ext", "MPL-2.0"),                               // cargo-miri (via `directories`)
@@ -108,6 +108,17 @@ const EXCEPTIONS: ExceptionList = &[
     // tidy-alphabetical-end
 ];
 
+/// These are exceptions to Rust's permissive licensing policy, and
+/// should be considered bugs. Exceptions are only allowed in Rust
+/// tooling. It is _crucial_ that no exception crates be dependencies
+/// of the Rust runtime (std/test).
+#[rustfmt::skip]
+const EXCEPTIONS_STDLIB: ExceptionList = &[
+    // tidy-alphabetical-start
+    ("fortanix-sgx-abi", "MPL-2.0"), // libstd but only for `sgx` target. FIXME: this dependency violates the documentation comment above.
+    // tidy-alphabetical-end
+];
+
 // FIXME uncomment once rust-lang/stdarch#1462 lands
 /*
 const EXCEPTIONS_STDARCH: ExceptionList = &[
@@ -228,10 +239,6 @@ const EXCEPTIONS_NON_STANDARD_LICENSE_DEPS: &[&str] = &[
     "ring",
 ];
 
-/// These are the root crates that are part of the runtime. The licenses for
-/// these and all their dependencies *must not* be in the exception list.
-const RUNTIME_CRATES: &[&str] = &["std", "core", "alloc", "test", "panic_abort", "panic_unwind"];
-
 const PERMITTED_DEPS_LOCATION: &str = concat!(file!(), ":", line!());
 
 /// Crates rustc is allowed to depend on. Avoid adding to the list if possible.
@@ -240,7 +247,6 @@ const PERMITTED_DEPS_LOCATION: &str = concat!(file!(), ":", line!());
 /// rustc. Please check with the compiler team before adding an entry.
 const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     // tidy-alphabetical-start
-    "addr2line",
     "adler",
     "ahash",
     "aho-corasick",
@@ -256,7 +262,6 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     "cc",
     "cfg-if",
     "cfg_aliases",
-    "compiler_builtins",
     "cpufeatures",
     "crc32fast",
     "crossbeam-channel",
@@ -276,7 +281,6 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     "digest",
     "displaydoc",
     "dissimilar",
-    "dlmalloc",
     "either",
     "elsa",
     "ena",
@@ -291,7 +295,6 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     "fluent-langneg",
     "fluent-syntax",
     "fnv",
-    "fortanix-sgx-abi",
     "generic-array",
     "getopts",
     "getrandom",
@@ -354,12 +357,9 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     "pulldown-cmark-escape",
     "punycode",
     "quote",
-    "r-efi",
-    "r-efi-alloc",
     "rand",
     "rand_chacha",
     "rand_core",
-    "rand_xorshift",
     "rand_xoshiro",
     "redox_syscall",
     "regex",
@@ -429,7 +429,6 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     "unicode-security",
     "unicode-width",
     "unicode-xid",
-    "unwinding",
     "valuable",
     "version_check",
     "wasi",
@@ -463,6 +462,46 @@ const PERMITTED_RUSTC_DEPENDENCIES: &[&str] = &[
     // tidy-alphabetical-end
 ];
 
+const PERMITTED_STDLIB_DEPENDENCIES: &[&str] = &[
+    // tidy-alphabetical-start
+    "addr2line",
+    "adler",
+    "allocator-api2",
+    "cc",
+    "cfg-if",
+    "compiler_builtins",
+    "dlmalloc",
+    "fortanix-sgx-abi",
+    "getopts",
+    "gimli",
+    "hashbrown",
+    "hermit-abi",
+    "libc",
+    "memchr",
+    "miniz_oxide",
+    "object",
+    "r-efi",
+    "r-efi-alloc",
+    "rand",
+    "rand_core",
+    "rand_xorshift",
+    "rustc-demangle",
+    "unicode-width",
+    "unwinding",
+    "wasi",
+    "windows-sys",
+    "windows-targets",
+    "windows_aarch64_gnullvm",
+    "windows_aarch64_msvc",
+    "windows_i686_gnu",
+    "windows_i686_gnullvm",
+    "windows_i686_msvc",
+    "windows_x86_64_gnu",
+    "windows_x86_64_gnullvm",
+    "windows_x86_64_msvc",
+    // tidy-alphabetical-end
+];
+
 const PERMITTED_CRANELIFT_DEPENDENCIES: &[&str] = &[
     // tidy-alphabetical-start
     "ahash",
@@ -556,9 +595,8 @@ pub fn check(root: &Path, cargo: &Path, bad: &mut bool) {
             check_permitted_dependencies(&metadata, workspace, permitted_deps, crates, bad);
         }
 
-        if workspace == "." {
-            let runtime_ids = compute_runtime_crates(&metadata);
-            check_runtime_license_exceptions(&metadata, runtime_ids, bad);
+        if workspace == "library" {
+            check_runtime_license_exceptions(&metadata, bad);
             checked_runtime_licenses = true;
         }
     }
@@ -583,16 +621,8 @@ pub fn has_missing_submodule(root: &Path, submodules: &[&str]) -> bool {
 ///
 /// Unlike for tools we don't allow exceptions to the `LICENSES` list for the runtime with the sole
 /// exception of `fortanix-sgx-abi` which is only used on x86_64-fortanix-unknown-sgx.
-fn check_runtime_license_exceptions(
-    metadata: &Metadata,
-    runtime_ids: HashSet<&PackageId>,
-    bad: &mut bool,
-) {
+fn check_runtime_license_exceptions(metadata: &Metadata, bad: &mut bool) {
     for pkg in &metadata.packages {
-        if !runtime_ids.contains(&pkg.id) {
-            // Only checking dependencies of runtime libraries here.
-            continue;
-        }
         if pkg.source.is_none() {
             // No need to check local packages.
             continue;
@@ -613,20 +643,6 @@ fn check_runtime_license_exceptions(
                 continue;
             }
 
-            // This exception is due to the fact that the feature set of the
-            // `object` crate is different between rustc and libstd. In the
-            // standard library only a conservative set of features are enabled
-            // which notably does not include the `wasm` feature which pulls in
-            // this dependency. In the compiler, however, the `wasm` feature is
-            // enabled. This exception is intended to be here so long as the
-            // `EXCEPTIONS` above contains `wasmparser`, but once that goes away
-            // this can be removed.
-            if pkg.name == "wasmparser"
-                && pkg.license.as_deref() == Some("Apache-2.0 WITH LLVM-exception")
-            {
-                continue;
-            }
-
             tidy_error!(bad, "invalid license `{}` in `{}`", license, pkg.id);
         }
     }
@@ -758,16 +774,6 @@ fn pkg_from_id<'a>(metadata: &'a Metadata, id: &PackageId) -> &'a Package {
     metadata.packages.iter().find(|p| &p.id == id).unwrap()
 }
 
-/// Finds all the packages that are in the rust runtime.
-fn compute_runtime_crates<'a>(metadata: &'a Metadata) -> HashSet<&'a PackageId> {
-    let mut result = HashSet::new();
-    for name in RUNTIME_CRATES {
-        let id = &pkg_from_name(metadata, name).id;
-        deps_of(metadata, id, &mut result);
-    }
-    result
-}
-
 /// Recursively find all dependencies.
 fn deps_of<'a>(metadata: &'a Metadata, pkg_id: &'a PackageId, result: &mut HashSet<&'a PackageId>) {
     if !result.insert(pkg_id) {

src/bootstrap/src/core/metadata.rs

@@ -4,11 +4,6 @@ use std::path::Path;
 
 use crate::walk::{filter_dirs, walk};
 
-fn is_edition_2021(mut line: &str) -> bool {
-    line = line.trim();
-    line == "edition = \"2021\""
-}
-
 pub fn check(path: &Path, bad: &mut bool) {
     walk(path, |path, _is_dir| filter_dirs(path), &mut |entry, contents| {
         let file = entry.path();
@@ -17,8 +12,15 @@ pub fn check(path: &Path, bad: &mut bool) {
             return;
         }
 
-        let is_2021 = contents.lines().any(is_edition_2021);
-        if !is_2021 {
+        let is_2021 = contents.lines().any(|line| line.trim() == "edition = \"2021\"");
+
+        let is_workspace = contents.lines().any(|line| line.trim() == "[workspace]");
+        let is_package = contents.lines().any(|line| line.trim() == "[package]");
+        assert!(is_workspace || is_package);
+
+        // Check that all packages use the 2021 edition. Virtual workspaces don't allow setting an
+        // edition, so these shouldn't be checked.
+        if is_package && !is_2021 {
             tidy_error!(
                 bad,
                 "{} doesn't have `edition = \"2021\"` on a separate line",

src/tools/tidy/src/deps.rs

@@ -1,4 +1,4 @@
-thread 'main' panicked at library/core/src/panicking.rs:$LINE:$COL:
+thread 'main' panicked at core/src/panicking.rs:$LINE:$COL:
 attempted to compute the size or alignment of extern type `Opaque`
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread caused non-unwinding panic. aborting.

src/tools/tidy/src/edition.rs

@@ -1,4 +1,4 @@
-thread 'main' panicked at library/core/src/panicking.rs:$LINE:$COL:
+thread 'main' panicked at core/src/panicking.rs:$LINE:$COL:
 attempted to compute the size or alignment of extern type `A`
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread caused non-unwinding panic. aborting.

tests/ui/extern/extern-types-field-offset.run.stderr

@@ -1,4 +1,4 @@
-thread 'main' panicked at library/core/src/panicking.rs:$LINE:$COL:
+thread 'main' panicked at core/src/panicking.rs:$LINE:$COL:
 attempted to compute the size or alignment of extern type `A`
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread caused non-unwinding panic. aborting.

tests/ui/extern/extern-types-size_of_val.align.run.stderr

@@ -1,3 +1,3 @@
-thread 'main' panicked at library/alloc/src/raw_vec.rs:LL:CC:
+thread 'main' panicked at alloc/src/raw_vec.rs:LL:CC:
 capacity overflow
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

tests/ui/extern/extern-types-size_of_val.size.run.stderr

@@ -4,6 +4,6 @@ note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread 'main' panicked at $DIR/panic-in-cleanup.rs:16:9:
 BOOM
 stack backtrace:
-thread 'main' panicked at library/core/src/panicking.rs:$LINE:$COL:
+thread 'main' panicked at core/src/panicking.rs:$LINE:$COL:
 panic in a destructor during cleanup
 thread caused non-unwinding panic. aborting.

tests/ui/hygiene/panic-location.run.stderr

@@ -1,7 +1,7 @@
 thread 'main' panicked at $DIR/panic-in-ffi.rs:12:5:
 Test
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
-thread 'main' panicked at library/core/src/panicking.rs:$LINE:$COL:
+thread 'main' panicked at core/src/panicking.rs:$LINE:$COL:
 panic in a function that cannot unwind
 stack backtrace:
 thread caused non-unwinding panic. aborting.

tests/ui/panics/panic-in-cleanup.run.stderr

@@ -1,3 +1,3 @@
-thread 'main' panicked at library/std/src/io/stdio.rs:LL:CC:
+thread 'main' panicked at std/src/io/stdio.rs:LL:CC:
 failed printing to stdout: Broken pipe (os error 32)
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace