Skip to content

Rework Rustbuild to an eagerly compiling approach #43059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 51 commits into from
Jul 22, 2017
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
0a1b5e8
Move rule configs out of step
Mark-Simulacrum Jul 4, 2017
001e9f3
Move code into Step trait implementations.
Mark-Simulacrum Jul 5, 2017
6b3413d
Change code to work with the new system
Mark-Simulacrum Jul 5, 2017
cd3dd80
Add Builder and Step definitions.
Mark-Simulacrum Jul 5, 2017
6038830
Fixes warnings and errors introduced while moving code around
Mark-Simulacrum Jul 5, 2017
1a844eb
Move tool_cmd to tool.rs
Mark-Simulacrum Jul 5, 2017
276090e
Fix tool_cmd
Mark-Simulacrum Jul 5, 2017
7db49fb
Move cargo() to Builder
Mark-Simulacrum Jul 5, 2017
c114fe5
Finish fixing warnings and errors. Bootstrap builds.
Mark-Simulacrum Jul 5, 2017
ef1d1bd
Move code out of macro and into generic method.
Mark-Simulacrum Jul 6, 2017
e62fdf3
Pacify tidy
Mark-Simulacrum Jul 6, 2017
8264e42
Add documentation to Step and related methods on Builder.
Mark-Simulacrum Jul 7, 2017
b881aae
Document the process more thoroughly
Mark-Simulacrum Jul 4, 2017
1ab8930
Move compiletest config into a struct
Mark-Simulacrum Jul 7, 2017
aa8b93b
Rework compiletest implementation.
Mark-Simulacrum Jul 7, 2017
d812d43
Fix a nit.
Mark-Simulacrum Jul 7, 2017
d360af4
Migrate to serde_json entirely
Mark-Simulacrum Jul 4, 2017
a1fa268
Update to toml 0.4
Mark-Simulacrum Jul 4, 2017
c7435b5
Cherry pick changes from ce3abc5801f94292be9bc5fbe00b52f1ccb28672.
Mark-Simulacrum Jul 8, 2017
a5ab2ce
Fix a few errors introduced during rebase.
Mark-Simulacrum Jul 12, 2017
ceecd62
Fix more incorrectly transitioned code
Mark-Simulacrum Jul 12, 2017
5984e70
Cleanups and fixes throughout
Mark-Simulacrum Jul 13, 2017
6a85193
Clean up install
Mark-Simulacrum Jul 13, 2017
e7b1a60
Remove core_intrinsics feature gate
Mark-Simulacrum Jul 13, 2017
528646e
Utilize interning to allow Copy/Clone steps
Mark-Simulacrum Jul 14, 2017
270d1d6
Make the book default.
Mark-Simulacrum Jul 14, 2017
681b123
Require should_run to be implemented.
Mark-Simulacrum Jul 14, 2017
fc3d06a
Implement keep-stage support
Mark-Simulacrum Jul 14, 2017
17f4b8f
Remove outdated FIXME from cache
Mark-Simulacrum Jul 14, 2017
f19728f
Fix tidy
Mark-Simulacrum Jul 14, 2017
828b661
fail in case nothing to run was found
GuillaumeGomez Jul 10, 2017
dec44b0
Resolve rebase errors
Mark-Simulacrum Jul 17, 2017
28defe0
Remove deserialize
aidanhs Jul 16, 2017
7414868
Remove TypeId from stack in Builder
Mark-Simulacrum Jul 18, 2017
ba1fc82
Remove outdated code.
Mark-Simulacrum Jul 18, 2017
981afa5
Krate -> Crate
Mark-Simulacrum Jul 18, 2017
bca1e2f
Use a single line for serde annotations.
Mark-Simulacrum Jul 18, 2017
56128fb
Implement available paths list.
Mark-Simulacrum Jul 19, 2017
5bdec80
Allow iterating over step descriptions.
Mark-Simulacrum Jul 19, 2017
4a21c72
Fix a few issues found by comparing past/present
Mark-Simulacrum Jul 20, 2017
f1d04a3
Don't run host-only tests when targeting another platform
Mark-Simulacrum Jul 20, 2017
951616c
Don't include lldb/gdb in default tests
Mark-Simulacrum Jul 20, 2017
bcd73c9
Change default documentation rules to correspond with previous state.
Mark-Simulacrum Jul 20, 2017
8563280
Check RLS tests
Mark-Simulacrum Jul 20, 2017
e9c2242
Remove step.rs comments
Mark-Simulacrum Jul 20, 2017
d8aecc1
Remove step.rs
Mark-Simulacrum Jul 20, 2017
b05af49
Add an optional condition to constrain defaults.
Mark-Simulacrum Jul 20, 2017
6a67a05
Change make_run signature to taking a RunConfig struct for refactorab…
Mark-Simulacrum Jul 20, 2017
d302c18
Fix StartupObject build
Mark-Simulacrum Jul 22, 2017
8f2e576
Add make_run to distcheck.
Mark-Simulacrum Jul 22, 2017
1c11823
Make distcheck work again.
Mark-Simulacrum Jul 22, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Allow iterating over step descriptions.
This simplifies code and allows working mostly with normal Rust instead
of macros.
  • Loading branch information
Mark-Simulacrum committed Jul 20, 2017
commit 5bdec80fe3989321c85545b4d6ee7719f1f568cb
220 changes: 113 additions & 107 deletions src/bootstrap/builder.rs
Original file line number Diff line number Diff line change
@@ -93,6 +93,89 @@ pub trait Step: 'static + Clone + Debug + PartialEq + Eq + Hash {
}
}

struct StepDescription {
default: bool,
only_hosts: bool,
only_build_targets: bool,
only_build: bool,
should_run: fn(ShouldRun) -> ShouldRun,
make_run: fn(&Builder, Option<&Path>, Interned<String>, Interned<String>),
}

impl StepDescription {
fn from<S: Step>() -> StepDescription {
StepDescription {
default: S::DEFAULT,
only_hosts: S::ONLY_HOSTS,
only_build_targets: S::ONLY_BUILD_TARGETS,
only_build: S::ONLY_BUILD,
should_run: S::should_run,
make_run: S::make_run,
}
}

fn maybe_run(&self, builder: &Builder, path: Option<&Path>) {
let build = builder.build;
let hosts = if self.only_build_targets || self.only_build {
&build.config.host[..1]
} else {
&build.hosts
};

// Determine the actual targets participating in this rule.
// NOTE: We should keep the full projection from build triple to
// the hosts for the dist steps, now that the hosts array above is
// truncated to avoid duplication of work in that case. Therefore
// the original non-shadowed hosts array is used below.
let targets = if self.only_hosts {
// If --target was specified but --host wasn't specified,
// don't run any host-only tests. Also, respect any `--host`
// overrides as done for `hosts`.
if build.flags.host.len() > 0 {
&build.flags.host[..]
} else if build.flags.target.len() > 0 {
&[]
} else if self.only_build {
&build.config.host[..1]
} else {
&build.config.host[..]
}
} else {
&build.targets
};

for host in hosts {
for target in targets {
(self.make_run)(builder, path, *host, *target);
}
}
}

fn run(v: &[StepDescription], builder: &Builder, paths: &[PathBuf]) {
if paths.is_empty() {
for desc in v {
if desc.default {
desc.maybe_run(builder, None);
}
}
} else {
for path in paths {
let mut attempted_run = false;
for desc in v {
if (desc.should_run)(ShouldRun::new(builder)).run(path) {
attempted_run = true;
desc.maybe_run(builder, Some(path));
}
}

if !attempted_run {
eprintln!("Warning: no rules matched {}.", path.display());
}
}
}
}
}

#[derive(Clone)]
pub struct ShouldRun<'a> {
builder: &'a Builder<'a>,
@@ -140,33 +223,34 @@ pub enum Kind {
Install,
}

macro_rules! check {
($self:ident, $paths:ident, $($rule:ty),+ $(,)*) => {{
let paths = $paths;
if paths.is_empty() {
$({
if <$rule>::DEFAULT {
$self.maybe_run::<$rule>(None);
}
})+
} else {
for path in paths {
let mut attempted_run = false;
$({
if <$rule>::should_run(ShouldRun::new($self)).run(path) {
attempted_run = true;
$self.maybe_run::<$rule>(Some(path));
}
})+
if !attempted_run {
eprintln!("Warning: no rules matched {}.", path.display());
}
}
impl<'a> Builder<'a> {
fn get_step_descriptions(kind: Kind) -> Vec<StepDescription> {
macro_rules! describe {
($($rule:ty),+ $(,)*) => {{
vec![$(StepDescription::from::<$rule>()),+]
}};
}
}};
}
match kind {
Kind::Build => describe!(compile::Std, compile::Test, compile::Rustc,
compile::StartupObjects, tool::BuildManifest, tool::Rustbook, tool::ErrorIndex,
tool::UnstableBookGen, tool::Tidy, tool::Linkchecker, tool::CargoTest,
tool::Compiletest, tool::RemoteTestServer, tool::RemoteTestClient,
tool::RustInstaller, tool::Cargo, tool::Rls),
Kind::Test => describe!(check::Tidy, check::Bootstrap, check::Compiletest, check::Crate,
check::CrateLibrustc, check::Linkcheck, check::Cargotest, check::Cargo, check::Docs,
check::ErrorIndex, check::Distcheck),
Kind::Bench => describe!(check::Crate, check::CrateLibrustc),
Kind::Doc => describe!(doc::UnstableBook, doc::UnstableBookGen, doc::TheBook,
doc::Standalone, doc::Std, doc::Test, doc::Rustc, doc::ErrorIndex, doc::Nomicon,
doc::Reference),
Kind::Dist => describe!(dist::Docs, dist::Mingw, dist::Rustc, dist::DebuggerScripts,
dist::Std, dist::Analysis, dist::Src, dist::PlainSourceTarball, dist::Cargo,
dist::Rls, dist::Extended, dist::HashSign),
Kind::Install => describe!(install::Docs, install::Std, install::Cargo, install::Rls,
install::Analysis, install::Src, install::Rustc),
}
}

impl<'a> Builder<'a> {
pub fn get_help(build: &Build, subcommand: &str) -> Option<String> {
let kind = match subcommand {
"build" => Kind::Build,
@@ -188,31 +272,8 @@ impl<'a> Builder<'a> {

let builder = &builder;
let mut should_run = ShouldRun::new(builder);
macro_rules! into_shouldrun {
($should_run:ident, $($rule:ty),+ $(,)*) => {{
$(
$should_run = <$rule>::should_run($should_run);
)+
}};
}
match builder.kind {
Kind::Build => into_shouldrun!(should_run, compile::Std, compile::Test, compile::Rustc,
compile::StartupObjects, tool::BuildManifest, tool::Rustbook, tool::ErrorIndex,
tool::UnstableBookGen, tool::Tidy, tool::Linkchecker, tool::CargoTest,
tool::Compiletest, tool::RemoteTestServer, tool::RemoteTestClient,
tool::RustInstaller, tool::Cargo, tool::Rls),
Kind::Test => into_shouldrun!(should_run, check::Tidy, check::Bootstrap,
check::Compiletest, check::Crate, check::CrateLibrustc, check::Linkcheck,
check::Cargotest, check::Cargo, check::Docs, check::ErrorIndex, check::Distcheck),
Kind::Bench => into_shouldrun!(should_run, check::Crate, check::CrateLibrustc),
Kind::Doc => into_shouldrun!(should_run, doc::UnstableBook, doc::UnstableBookGen,
doc::TheBook, doc::Standalone, doc::Std, doc::Test, doc::Rustc, doc::ErrorIndex,
doc::Nomicon, doc::Reference),
Kind::Dist => into_shouldrun!(should_run, dist::Docs, dist::Mingw, dist::Rustc,
dist::DebuggerScripts, dist::Std, dist::Analysis, dist::Src,
dist::PlainSourceTarball, dist::Cargo, dist::Rls, dist::Extended, dist::HashSign),
Kind::Install => into_shouldrun!(should_run, install::Docs, install::Std,
install::Cargo, install::Rls, install::Analysis, install::Src, install::Rustc),
for desc in Builder::get_step_descriptions(builder.kind) {
should_run = (desc.should_run)(should_run);
}
let mut help = String::from("Available paths:\n");
for path in should_run.paths {
@@ -240,30 +301,12 @@ impl<'a> Builder<'a> {
stack: RefCell::new(Vec::new()),
};

let builder = &builder;
match builder.kind {
Kind::Build => check!(builder, paths, compile::Std, compile::Test, compile::Rustc,
compile::StartupObjects, tool::BuildManifest, tool::Rustbook, tool::ErrorIndex,
tool::UnstableBookGen, tool::Tidy, tool::Linkchecker, tool::CargoTest,
tool::Compiletest, tool::RemoteTestServer, tool::RemoteTestClient,
tool::RustInstaller, tool::Cargo, tool::Rls),
Kind::Test => check!(builder, paths, check::Tidy, check::Bootstrap, check::Compiletest,
check::Crate, check::CrateLibrustc, check::Linkcheck, check::Cargotest,
check::Cargo, check::Docs, check::ErrorIndex, check::Distcheck),
Kind::Bench => check!(builder, paths, check::Crate, check::CrateLibrustc),
Kind::Doc => builder.default_doc(Some(paths)),
Kind::Dist => check!(builder, paths, dist::Docs, dist::Mingw, dist::Rustc,
dist::DebuggerScripts, dist::Std, dist::Analysis, dist::Src,
dist::PlainSourceTarball, dist::Cargo, dist::Rls, dist::Extended, dist::HashSign),
Kind::Install => check!(builder, paths, install::Docs, install::Std, install::Cargo,
install::Rls, install::Analysis, install::Src, install::Rustc),
}
StepDescription::run(&Builder::get_step_descriptions(builder.kind), &builder, paths);
}

pub fn default_doc(&self, paths: Option<&[PathBuf]>) {
let paths = paths.unwrap_or(&[]);
check!(self, paths, doc::UnstableBook, doc::UnstableBookGen, doc::TheBook, doc::Standalone,
doc::Std, doc::Test, doc::Rustc, doc::ErrorIndex, doc::Nomicon, doc::Reference);
StepDescription::run(&Builder::get_step_descriptions(Kind::Doc), self, paths);
}

/// Obtain a compiler at a given stage and for a given host. Explictly does
@@ -514,43 +557,6 @@ impl<'a> Builder<'a> {
cargo
}

fn maybe_run<S: Step>(&self, path: Option<&Path>) {
let build = self.build;
let hosts = if S::ONLY_BUILD_TARGETS || S::ONLY_BUILD {
&build.config.host[..1]
} else {
&build.hosts
};

// Determine the actual targets participating in this rule.
// NOTE: We should keep the full projection from build triple to
// the hosts for the dist steps, now that the hosts array above is
// truncated to avoid duplication of work in that case. Therefore
// the original non-shadowed hosts array is used below.
let targets = if S::ONLY_HOSTS {
// If --target was specified but --host wasn't specified,
// don't run any host-only tests. Also, respect any `--host`
// overrides as done for `hosts`.
if build.flags.host.len() > 0 {
&build.flags.host[..]
} else if build.flags.target.len() > 0 {
&[]
} else if S::ONLY_BUILD {
&build.config.host[..1]
} else {
&build.config.host[..]
}
} else {
&build.targets
};

for host in hosts {
for target in targets {
S::make_run(self, path, *host, *target);
}
}
}

/// Ensure that a given step is built, returning it's output. This will
/// cache the step, so it is safe (and good!) to call this as often as
/// needed to ensure that all dependencies are built.