Skip to content

Commit ed31bb0

Browse files
rickeylevrickeylev
committed
feat: Expose Python C headers through the toolchain.
This allows getting a build's `cc_library` of Python headers through toolchain resolution instead of having to use the underlying toolchain's repository `:python_headers` target directly. Without this feature, it's not possible to reliably and correctly get the C information about the runtime a build is going to use. Existing solutions require carefully setting up repo names, external state, and/or using specific build rules. In comparison, with this feature, consumers are able to simply ask for the current headers via a special target or manually lookup the toolchain and pull the relevant information; toolchain resolution handles finding the correct headers. The basic way this works is by registering a second toolchain to carry C/C++ related information; as such, it is named `py_cc_toolchain`. The py cc toolchain has the same constraint settings as the regular py toolchain; an expected invariant is that there is a 1:1 correspondence between the two. This base functionality allows a consuming rule implementation to use toolchain resolution to find the Python C toolchain information. Usually what downstream consumers need are the headers to feed into another `cc_library` (or equivalent) target, so, rather than have every project reimplement the same "lookup and forward cc_library info" logic, this is provided by the `//python/cc:current_py_cc_headers` target. Targets that need the headers can then depend on that target as if it was a `cc_library` target. Work towards bazel-contrib#824
1 parent 5b8fa22 commit ed31bb0

23 files changed

+767
-6
lines changed

‎.bazelrc

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
6+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
88

99
test --test_output=errors
1010

‎python/BUILD.bazel

+1
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ licenses(["notice"])
3333
filegroup(
3434
name = "distribution",
3535
srcs = glob(["**"]) + [
36+
"//python/cc:distribution",
3637
"//python/config_settings:distribution",
3738
"//python/constraints:distribution",
3839
"//python/private:distribution",

‎python/cc/BUILD.bazel

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Package for C/C++ specific functionality of the Python rules.
2+
3+
load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
4+
load("//python/private:current_py_cc_headers.bzl", "current_py_cc_headers")
5+
load("//python/private:util.bzl", "BZLMOD_ENABLED")
6+
7+
# This target provides the C headers for whatever the current toolchain is
8+
# for the consuming rule. It basically acts like a cc_library by forwarding
9+
# on the providers for the underlying cc_library that the toolchain is using.
10+
current_py_cc_headers(
11+
name = "current_py_cc_headers",
12+
# Building this directly will fail unless a py cc toolchain is registered,
13+
# and it's only under bzlmod that one is registered by default.
14+
tags = [] if BZLMOD_ENABLED else ["manual"],
15+
visibility = ["//visibility:public"],
16+
)
17+
18+
toolchain_type(
19+
name = "toolchain_type",
20+
visibility = ["//visibility:public"],
21+
)
22+
23+
filegroup(
24+
name = "distribution",
25+
srcs = glob(["**"]),
26+
visibility = ["//:__subpackages__"],
27+
)
28+
29+
bzl_library(
30+
name = "py_cc_toolchain_bzl",
31+
srcs = ["py_cc_toolchain.bzl"],
32+
deps = ["//python/private:py_cc_toolchain_bzl"],
33+
)
34+
35+
bzl_library(
36+
name = "py_cc_toolchain_info_bzl",
37+
srcs = ["py_cc_toolchain_info.bzl"],
38+
deps = ["//python/private:py_cc_toolchain_info_bzl"],
39+
)

‎python/cc/py_cc_toolchain.bzl

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Public entry point for py_cc_toolchain rule."""
16+
17+
load("//python/private:py_cc_toolchain_macro.bzl", _py_cc_toolchain = "py_cc_toolchain")
18+
19+
py_cc_toolchain = _py_cc_toolchain

‎python/cc/py_cc_toolchain_info.bzl

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Public entry point for PyCcToolchainInfo."""
16+
17+
load("//python/private:py_cc_toolchain_info.bzl", _PyCcToolchainInfo = "PyCcToolchainInfo")
18+
19+
PyCcToolchainInfo = _PyCcToolchainInfo

‎python/private/BUILD.bazel

+17
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,23 @@ bzl_library(
5151
deps = ["@bazel_skylib//lib:types"],
5252
)
5353

54+
bzl_library(
55+
name = "py_cc_toolchain_bzl",
56+
srcs = [
57+
"py_cc_toolchain_macro.bzl",
58+
"py_cc_toolchain_rule.bzl",
59+
],
60+
deps = [
61+
":py_cc_toolchain_info_bzl",
62+
":util_bzl",
63+
],
64+
)
65+
66+
bzl_library(
67+
name = "py_cc_toolchain_info_bzl",
68+
srcs = ["py_cc_toolchain_info.bzl"],
69+
)
70+
5471
# @bazel_tools can't define bzl_library itself, so we just put a wrapper around it.
5572
bzl_library(
5673
name = "bazel_tools_bzl",

‎python/private/current_py_cc_headers.bzl

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of current_py_cc_headers rule."""
16+
17+
def _current_py_cc_headers_impl(ctx):
18+
py_cc_toolchain = ctx.toolchains["//python/cc:toolchain_type"].py_cc_toolchain
19+
return py_cc_toolchain.headers.providers_map.values()
20+
21+
current_py_cc_headers = rule(
22+
implementation = _current_py_cc_headers_impl,
23+
toolchains = ["//python/cc:toolchain_type"],
24+
provides = [CcInfo],
25+
)

‎python/private/py_cc_toolchain_info.bzl

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of PyCcToolchainInfo."""
16+
17+
PyCcToolchainInfo = provider(
18+
doc = "C/C++ information about the Python runtime.",
19+
fields = {
20+
"headers": """\
21+
(struct) Information about the header files, with fields:
22+
* providers_map: a dict of string to provider instances. The key should be
23+
a fully qualified name (e.g. `@rules_foo//bar:baz.bzl#MyInfo`) of the
24+
provider to uniquely identify its type.
25+
26+
The following keys are always present:
27+
* CcInfo: the CcInfo provider instance for the headers.
28+
* DefaultInfo: the DefaultInfo provider instance for the headers.
29+
30+
A map is used to allow additional providers from the originating headers
31+
target (typically a `cc_library`) to be propagated to consumers (directly
32+
exposing a Target object can cause memory issues and is an anti-pattern).
33+
34+
When consuming this map, it's suggested to use `providers_map.values()` to
35+
return all providers; or copy the map and filter out or replace keys as
36+
appropriate. Note that any keys begining with `_` (underscore) are
37+
considered private and should be forward along as-is (this better allows
38+
e.g. `:current_py_cc_headers` to act as the underlying headers target it
39+
represents).
40+
""",
41+
"python_version": "(str) The Python Major.Minor version.",
42+
},
43+
)

‎python/private/py_cc_toolchain_macro.bzl

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Fronting macro for the py_cc_toolchain rule."""
16+
17+
load(":py_cc_toolchain_rule.bzl", _py_cc_toolchain = "py_cc_toolchain")
18+
load(":util.bzl", "add_tag")
19+
20+
# A fronting macro is used because macros have user-observable behavior;
21+
# using one from the onset avoids introducing those changes in the future.
22+
def py_cc_toolchain(**kwargs):
23+
# This tag is added to easily identify usages through other macros.
24+
add_tag(kwargs, "@rules_python//python:py_cc_toolchain")
25+
_py_cc_toolchain(**kwargs)

‎python/private/py_cc_toolchain_rule.bzl

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Implementation of py_cc_toolchain rule."""
16+
17+
load(":py_cc_toolchain_info.bzl", "PyCcToolchainInfo")
18+
19+
def _py_cc_toolchain_impl(ctx):
20+
py_cc_toolchain = PyCcToolchainInfo(
21+
headers = struct(
22+
providers_map = {
23+
"CcInfo": ctx.attr.headers[CcInfo],
24+
"DefaultInfo": ctx.attr.headers[DefaultInfo],
25+
},
26+
),
27+
python_version = ctx.attr.python_version,
28+
)
29+
return [platform_common.ToolchainInfo(
30+
py_cc_toolchain = py_cc_toolchain,
31+
)]
32+
33+
py_cc_toolchain = rule(
34+
implementation = _py_cc_toolchain_impl,
35+
attrs = {
36+
"headers": attr.label(
37+
doc = ("Target that provides the Python headers. Typically this " +
38+
"is a cc_library target."),
39+
providers = [CcInfo],
40+
mandatory = True,
41+
),
42+
"python_version": attr.string(
43+
doc = "The Major.minor Python version, e.g. 3.11",
44+
mandatory = True,
45+
),
46+
},
47+
)

‎python/private/toolchains_repo.bzl

+9
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,15 @@ toolchain(
8383
toolchain = "@{user_repository_name}_{platform}//:python_runtimes",
8484
toolchain_type = "@bazel_tools//tools/python:toolchain_type",
8585
)
86+
87+
toolchain(
88+
name = "{prefix}{platform}_py_cc_toolchain",
89+
target_compatible_with = {compatible_with},
90+
target_settings = {target_settings},
91+
toolchain = "@{user_repository_name}_{platform}//:py_cc_toolchain",
92+
toolchain_type = "@rules_python//python/cc:toolchain_type",
93+
94+
)
8695
""".format(
8796
compatible_with = meta.compatible_with,
8897
platform = platform,

‎python/private/util.bzl

+33-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,25 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
115
"""Functionality shared by multiple pieces of code."""
216

317
load("@bazel_skylib//lib:types.bzl", "types")
418

19+
# When bzlmod is enabled, canonical repos names have @@ in them, while under
20+
# workspace builds, there is never a @@ in labels.
21+
BZLMOD_ENABLED = "@@" in str(Label("//:unused"))
22+
523
def copy_propagating_kwargs(from_kwargs, into_kwargs = None):
624
"""Copies args that must be compatible between two targets with a dependency relationship.
725
@@ -46,15 +64,26 @@ def add_migration_tag(attrs):
4664
Returns:
4765
The same `attrs` object, but modified.
4866
"""
67+
add_tag(attrs, _MIGRATION_TAG)
68+
return attrs
69+
70+
def add_tag(attrs, tag):
71+
"""Adds `tag` to `attrs["tags"]`.
72+
73+
Args:
74+
attrs: dict of keyword args. It is modified in place.
75+
tag: str, the tag to add.
76+
"""
4977
if "tags" in attrs and attrs["tags"] != None:
5078
tags = attrs["tags"]
5179

5280
# Preserve the input type: this allows a test verifying the underlying
5381
# rule can accept the tuple for the tags argument.
5482
if types.is_tuple(tags):
55-
attrs["tags"] = tags + (_MIGRATION_TAG,)
83+
attrs["tags"] = tags + (tag,)
5684
else:
57-
attrs["tags"] = tags + [_MIGRATION_TAG]
85+
# List concatenation is necessary because the original value
86+
# may be a frozen list.
87+
attrs["tags"] = tags + [tag]
5888
else:
59-
attrs["tags"] = [_MIGRATION_TAG]
60-
return attrs
89+
attrs["tags"] = [tag]

‎python/repositories.bzl

+7
Original file line numberDiff line numberDiff line change
@@ -265,6 +265,7 @@ def _python_repository_impl(rctx):
265265
# Generated by python/repositories.bzl
266266
267267
load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair")
268+
load("@rules_python//python/cc:py_cc_toolchain.bzl", "py_cc_toolchain")
268269
269270
package(default_visibility = ["//visibility:public"])
270271
@@ -336,6 +337,12 @@ py_runtime_pair(
336337
py2_runtime = None,
337338
py3_runtime = ":py3_runtime",
338339
)
340+
341+
py_cc_toolchain(
342+
name = "py_cc_toolchain",
343+
headers = ":python_headers",
344+
python_version = "{python_version}",
345+
)
339346
""".format(
340347
glob_exclude = repr(glob_exclude),
341348
glob_include = repr(glob_include),

‎tests/BUILD.bazel

+2
Original file line numberDiff line numberDiff line change
@@ -28,5 +28,7 @@ build_test(
2828
"//python:py_runtime_info_bzl",
2929
"//python:py_runtime_pair_bzl",
3030
"//python:py_test_bzl",
31+
"//python/cc:py_cc_toolchain_bzl",
32+
"//python/cc:py_cc_toolchain_info_bzl",
3133
],
3234
)

0 commit comments

Comments
 (0)