Skip to content

Commit 727508e

Browse files
committed
Standardise on pip_parse
1 parent 6a43ebd commit 727508e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+247
-810
lines changed

.bazelci/presubmit.yml

-1
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,5 @@ platforms:
3636
- "-//gazelle/..."
3737
# The dependencies needed for this test are not cross-platform: https://github.com/bazelbuild/rules_python/issues/260
3838
- "-//tests:pip_repository_entry_points_example"
39-
- "-//tests:pip_deps_example"
4039
test_flags:
4140
- "--test_tag_filters=-fix-windows"

README.md

+25-53
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@
77

88
This repository is the home of the core Python rules -- `py_library`,
99
`py_binary`, `py_test`, and related symbols that provide the basis for Python
10-
support in Bazel. It also contains packaging rules for integrating with PyPI
11-
(`pip`). Documentation lives in the
10+
support in Bazel. It also contains package installation rules for integrating with PyPI and other package indices. Documentation lives in the
1211
[`docs/`](https://github.com/bazelbuild/rules_python/tree/main/docs)
1312
directory and in the
1413
[Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html).
@@ -24,7 +23,7 @@ Once they are fully migrated to rules_python, they may evolve at a different
2423
rate, but this repository will still follow
2524
[semantic versioning](https://semver.org).
2625

27-
The packaging rules (`pip_install`, etc.) are less stable. We may make breaking
26+
The package installation rules (`pip_install`, `pip_parse` etc.) are less stable. We may make breaking
2827
changes as they evolve.
2928

3029
This repository is maintained by the Bazel community. Neither Google, nor the
@@ -101,70 +100,27 @@ py_binary(
101100
)
102101
```
103102

104-
## Using the packaging rules
103+
## Using the package installation rules
105104

106105
Usage of the packaging rules involves two main steps.
107106

108-
1. [Installing `pip` dependencies](#installing-pip-dependencies)
109-
2. [Consuming `pip` dependencies](#consuming-pip-dependencies)
107+
1. [Installing third_party packages](#installing-third_party-packages)
108+
2. [Using third_party packages as dependencies](#using-third_party-packages-as-dependencies)
110109

111-
The packaging rules create two kinds of repositories: A central external repo that holds
110+
The package installation rules create two kinds of repositories: A central external repo that holds
112111
downloaded wheel files, and individual external repos for each wheel's extracted
113112
contents. Users only need to interact with the central external repo; the wheel repos
114113
are essentially an implementation detail. The central external repo provides a
115114
`WORKSPACE` macro to create the wheel repos, as well as a function, `requirement()`, for use in
116115
`BUILD` files that translates a pip package name into the label of a `py_library`
117116
target in the appropriate wheel repo.
118117

119-
### Installing `pip` dependencies
118+
### Installing third_party packages
120119

121120
To add pip dependencies to your `WORKSPACE`, load the `pip_install` function, and call it to create the
122121
central external repo and individual wheel external repos.
123122

124123

125-
```python
126-
load("@rules_python//python:pip.bzl", "pip_install")
127-
128-
# Create a central external repo, @my_deps, that contains Bazel targets for all the
129-
# third-party packages specified in the requirements.txt file.
130-
pip_install(
131-
name = "my_deps",
132-
requirements = "//path/to:requirements.txt",
133-
)
134-
```
135-
136-
Note that since `pip_install` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no
137-
information about the Python toolchain and cannot enforce that the interpreter
138-
used to invoke pip matches the interpreter used to run `py_binary` targets. By
139-
default, `pip_install` uses the system command `"python3"`. This can be overridden by passing the
140-
`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_install`.
141-
142-
You can have multiple `pip_install`s in the same workspace. This will create multiple external repos that have no relation to
143-
one another, and may result in downloading the same wheels multiple times.
144-
145-
As with any repository rule, if you would like to ensure that `pip_install` is
146-
re-executed in order to pick up a non-hermetic change to your environment (e.g.,
147-
updating your system `python` interpreter), you can force it to re-execute by running
148-
`bazel sync --only [pip_install name]`.
149-
150-
### Fetch `pip` dependencies lazily
151-
152-
One pain point with `pip_install` is the need to download all dependencies resolved by
153-
your requirements.txt before the bazel analysis phase can start. For large python monorepos
154-
this can take a long time, especially on slow connections.
155-
156-
`pip_parse` provides a solution to this problem. If you can provide a lock
157-
file of all your python dependencies `pip_parse` will translate each requirement into its own external repository.
158-
Bazel will only fetch/build wheels for the requirements in the subgraph of your build target.
159-
160-
There are API differences between `pip_parse` and `pip_install`:
161-
1. `pip_parse` requires a fully resolved lock file of your python dependencies. You can generate this by using the `compile_pip_requirements` rule,
162-
running `pip-compile` directly, or using virtualenv and `pip freeze`. `pip_parse` uses a label argument called `requirements_lock` instead of
163-
`requirements` to make this distinction clear.
164-
2. `pip_parse` translates your requirements into a starlark macro called `install_deps`. You must call this macro in your WORKSPACE to
165-
declare your dependencies.
166-
167-
168124
```python
169125
load("@rules_python//python:pip.bzl", "pip_parse")
170126

@@ -174,14 +130,30 @@ pip_parse(
174130
name = "my_deps",
175131
requirements_lock = "//path/to:requirements_lock.txt",
176132
)
177-
178133
# Load the starlark macro which will define your dependencies.
179134
load("@my_deps//:requirements.bzl", "install_deps")
180135
# Call it to define repos for your requirements.
181136
install_deps()
182137
```
183138

184-
### Consuming `pip` dependencies
139+
Note that since `pip_parse` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no
140+
information about the Python toolchain and cannot enforce that the interpreter
141+
used to invoke pip matches the interpreter used to run `py_binary` targets. By
142+
default, `pip_parse` uses the system command `"python3"`. This can be overridden by passing the
143+
`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_parse`.
144+
145+
You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to
146+
one another, and may result in downloading the same wheels multiple times.
147+
148+
As with any repository rule, if you would like to ensure that `pip_parse` is
149+
re-executed in order to pick up a non-hermetic change to your environment (e.g.,
150+
updating your system `python` interpreter), you can force it to re-execute by running
151+
`bazel sync --only [pip_parse name]`.
152+
153+
Note: The `pip_install` rule is deprecated. `pip_parse` offers identical functionality and both `pip_install`
154+
and `pip_parse` now have the same implementation. The name `pip_install` may be removed in a future version of the rules.
155+
156+
### Using third_party packages as dependencies
185157

186158
Each extracted wheel repo contains a `py_library` target representing
187159
the wheel's contents. There are two ways to access this library. The

docs/pip.md

+6-69
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)