docs: rules_python bzlmod GA and 1.0 prep (bazel-contrib#2296)

This is documenting the current state and closing the last remaining
TODO items for 1.0 release.

Work towards bazel-contrib#1361.

---------

Co-authored-by: Greg Roodt <[email protected]>
Co-authored-by: Richard Levasseur <[email protected]>

Author: 3 people

BZLMOD_SUPPORT.md

@@ -2,10 +2,12 @@
 
 ## `rules_python` `bzlmod` support
 
-- Status: Beta
+- Status: GA
 - Full Feature Parity: No
+    - `rules_python`: Yes
+    - `rules_python_gazelle_plugin`: No (see below).
 
-Some features are missing or broken, and the public APIs are not yet stable.
+In general `bzlmod` has more features than `WORKSPACE` and users are encouraged to migrate.
 
 ## Configuration
 
@@ -27,15 +29,6 @@ A user does not use `local_path_override` stanza and would define the version in
 
 A second example, in [examples/bzlmod_build_file_generation](examples/bzlmod_build_file_generation) demonstrates the use of `bzlmod` to configure `gazelle` support for `rules_python`.
 
-## Feature parity
-
-This rule set does not have full feature partity with the older `WORKSPACE` type configuration:
-
-1. Gazelle does not support finding deps in sub-modules.  For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
-2. We have some features that are still not fully flushed out, and the user interface may change.
-
-Check ["issues"](/bazelbuild/rules_python/issues) for an up to date list.
-
 ## Differences in behavior from WORKSPACE
 
 ### Default toolchain is not the local system Python
@@ -52,10 +45,35 @@ platforms.
 If you want to use the same toolchain as what WORKSPACE used, then manually
 register the builtin Bazel Python toolchain by doing
 `register_toolchains("@bazel_tools//tools/python:autodetecting_toolchain")`.
-**IMPORTANT: this should only be done in a root module, and may intefere with
+
+Note that using this builtin Bazel toolchain is deprecated and unsupported.
+See the {obj}`runtime_env_toolchains` docs for a replacement that is marginally
+better supported.
+**IMPORTANT: this should only be done in a root module, and may interfere with
 the toolchains rules_python registers**.
 
 NOTE: Regardless of your toolchain, due to
 [#691](https://github.com/bazelbuild/rules_python/issues/691), `rules_python`
 still relies on a local Python being available to bootstrap the program before
 handing over execution to the toolchain Python.
+
+To override this behaviour see {obj}`--bootstrap_impl=script`, which switches
+to `bash`-based bootstrap on UNIX systems.
+
+### Better PyPI package downloading on bzlmod
+
+On `bzlmod` users have the option to use the `bazel_downloader` to download packages
+and work correctly when `host` platform is not the same as the `target` platform. This
+provides faster package download times and integration with the credentials helper.
+
+### Extra targets in `whl_library` repos
+
+Due to how `bzlmod` is designed and the visibility rules that it enforces, it is best to use
+the targets in the `whl` repos as they do not rely on using the `annotations` API to
+add extra targets to so-called `spoke` repos. For alternatives that should cover most of the
+existing usecases please see:
+* {bzl:obj}`py_console_script_binary` to create `entry_point` targets.
+* {bzl:obj}`whl_filegroup` to extract filegroups from the `whl` targets (e.g. `@pip//numpy:whl`)
+* {bzl:obj}`pip.override` to patch the downloaded `whl` files. Using that you
+  can change the `METADATA` of the `whl` file that will influence how
+  `rules_python` code generation behaves.

CONTRIBUTING.md

@@ -194,8 +194,13 @@ The general process is:
    of. The API for the control mechanism can be removed in this release.
 
 Note that the `+1` and `+2` releases are just examples; the steps are not
-required to happen in immedially subsequent releases.
+required to happen in immediately subsequent releases.
 
+Once The first major version is released, the process will be:
+1. In `N.M.0` we introduce the new behaviour, but it is disabled by a feature flag.
+2. In `N.M+1.0` we may choose the behaviour to become the default if it is not too
+   disruptive.
+3. In `N+1.0.0` we get rid of the old behaviour.
 
 ### How to control breaking changes
 

docs/getting-started.md

@@ -1,7 +1,7 @@
 # Getting started
 
-This doc is a simplified guide to help get started started quickly. It provides
-a simplified introduction to having a working Python program for both bzlmod
+This doc is a simplified guide to help get started quickly. It provides
+a simplified introduction to having a working Python program for both `bzlmod`
 and the older way of using `WORKSPACE`.
 
 It assumes you have a `requirements.txt` file with your PyPI dependencies.
@@ -23,11 +23,11 @@ bazel_dep(name = "rules_python", version = "0.0.0")
 
 pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
 pip.parse(
-    hub_name = "my_deps",
+    hub_name = "pypi",
     python_version = "3.11",
     requirements_lock = "//:requirements.txt",
 )
-use_repo(pip, "my_deps")
+use_repo(pip, "pypi")
 ```
 
 ## Using a WORKSPACE file
@@ -38,19 +38,14 @@ using Bzlmod. Here is a simplified setup to download the prebuilt runtimes.
 ```starlark
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
-
-# Update the SHA and VERSION to the lastest version available here:
-# https://github.com/bazelbuild/rules_python/releases.
-
-SHA="84aec9e21cc56fbc7f1335035a71c850d1b9b5cc6ff497306f84cced9a769841"
-
-VERSION="0.23.1"
+# Update the snippet based on the latest release below
+# https://github.com/bazelbuild/rules_python/releases
 
 http_archive(
     name = "rules_python",
-    sha256 = SHA,
-    strip_prefix = "rules_python-{}".format(VERSION),
-    url = "https://github.com/bazelbuild/rules_python/releases/download/{}/rules_python-{}.tar.gz".format(VERSION,VERSION),
+    sha256 = "ca77768989a7f311186a29747e3e95c936a41dffac779aff6b443db22290d913",
+    strip_prefix = "rules_python-0.36.0",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.36.0/rules_python-0.36.0.tar.gz",
 )
 
 load("@rules_python//python:repositories.bzl", "py_repositories")
@@ -66,14 +61,12 @@ python_register_toolchains(
     python_version = "3.11",
 )
 
-load("@python_3_11//:defs.bzl", "interpreter")
-
 load("@rules_python//python:pip.bzl", "pip_parse")
 
 pip_parse(
-    ...
-    python_interpreter_target = interpreter,
-    ...
+    name = "pypi",
+    python_interpreter_target = "@python_3_11_host//:python",
+    requirements_lock = "//:requirements.txt",
 )
 ```
 
@@ -89,8 +82,8 @@ py_binary(
   name = "main",
   srcs = ["main.py"],
   deps = [
-      "@my_deps//foo",
-      "@my_deps//bar",
+      "@pypi//foo",
+      "@pypi//bar",
   ]
 )
 ```

docs/index.md

@@ -1,39 +1,73 @@
 # Python Rules for Bazel
 
-rules_python is the home of the core Python rules -- `py_library`,
-`py_binary`, `py_test`, `py_proto_library`, and related symbols that provide the basis for Python
-support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.
+`rules_python` is the home for 4 major components with varying maturity levels.
 
-Documentation for rules_python lives here and in the
-[Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html).
+:::{topic} Core rules
 
-Examples are in the {gh-path}`examples` directory.
+The core Python rules -- `py_library`, `py_binary`, `py_test`,
+`py_proto_library`, and related symbols that provide the basis for Python
+support in Bazel.
 
-Currently, the core rules build into the Bazel binary, and the symbols in this
-repository are simple aliases. However, we are migrating the rules to Starlark and removing them from the Bazel binary. Therefore, the future-proof way to depend on Python rules is via this repository. See
-{ref}`Migrating from the Bundled Rules` below.
+When using Bazel 6 (or earlier), the core rules are bundled into the Bazel binary, and the symbols
+in this repository are simple aliases. On Bazel 7 and above `rules_python` uses
+a separate Starlark implementation,
+see {ref}`Migrating from the Bundled Rules` below.
 
-The core rules are stable. Their implementation in Bazel is subject to Bazel's
-[backward compatibility policy](https://docs.bazel.build/versions/master/backward-compatibility.html).
-Once migrated to rules_python, they may evolve at a different
-rate, but this repository will still follow [semantic versioning](https://semver.org).
+Once rules_python 1.0 is released, they will follow
+[semantic versioning](https://semver.org) and the breaking change policy
+outlined in the [support](support) page.
 
-The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See
-{gh-path}`How to contribute <CONTRIBUTING.md>` for information on our development workflow.
+:::
 
-## Bzlmod support
+:::{topic} PyPI integration
 
-- Status: Beta
-- Full Feature Parity: No
+Package installation rules for integrating with PyPI and other SimpleAPI
+compatible indexes.
 
-See {gh-path}`Bzlmod support <BZLMOD_SUPPORT.md>` for more details
+These rules work and can be used in production, but the cross-platform building
+that supports pulling PyPI dependencies for a target platform that is different
+from the host platform is still in beta and the APIs that are subject to potential
+change are marked as `experimental`.
+
+:::
+
+:::{topic} Sphinxdocs
+
+`sphinxdocs` rules allow users to generate documentation using Sphinx powered by Bazel, with additional functionality for documenting
+Starlark and Bazel code.
+
+The functionality is exposed because other projects find it useful, but 
+it is available as is and **the semantic versioning and
+compatibility policy used by `rules_python` does not apply**. 
+
+:::
+
+:::{topic} Gazelle plugin
+
+`gazelle` plugin for generating `BUILD.bazel` files based on Python source
+code.
+
+This is available as is and the semantic versioning used by `rules_python` does
+not apply.
+
+:::
+
+The Bazel community maintains this repository. Neither Google nor the Bazel
+team provides support for the code. However, this repository is part of the
+test suite used to vet new Bazel releases. See {gh-path}`How to contribute
+<CONTRIBUTING.md>` for information on our development workflow.
+
+## Examples
+
+This documentation is an example of `sphinxdocs` rules and the rest of the
+components have examples in the {gh-path}`examples` directory.
 
 ## Migrating from the bundled rules
 
 The core rules are currently available in Bazel as built-in symbols, but this
 form is deprecated. Instead, you should depend on rules_python in your
-`WORKSPACE` file and load the Python rules from
-`@rules_python//python:defs.bzl`.
+`WORKSPACE` or `MODULE.bazel` file and load the Python rules from
+`@rules_python//python:defs.bzl` or load paths described in the API documentation.
 
 A [buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md)
 fix is available to automatically migrate `BUILD` and `.bzl` files to add the
@@ -44,13 +78,18 @@ appropriate `load()` statements and rewrite uses of `native.py_*`.
 buildifier --lint=fix --warnings=native-py <files>
 ```
 
-Currently, the `WORKSPACE` file needs to be updated manually as per [Getting
-started](getting-started).
+Currently, the `WORKSPACE` file needs to be updated manually as per 
+[Getting started](getting-started).
 
 Note that Starlark-defined bundled symbols underneath
 `@bazel_tools//tools/python` are also deprecated. These are not yet rewritten
 by buildifier.
 
+## Migrating to bzlmod
+
+See {gh-path}`Bzlmod support <BZLMOD_SUPPORT.md>` for any behaviour differences between
+`bzlmod` and `WORKSPACE`.
+
 
 ```{toctree}
 :hidden:

examples/bzlmod/MODULE.bazel.lock

@@ -39,7 +39,7 @@ def gazelle_python_manifest(
             manifest, meaning testing it is just as expensive as generating it,
             but modifying it is much less likely to result in a merge conflict.
         pip_repository_name: the name of the pip_install or pip_repository target.
-        pip_deps_repository_name: deprecated - the old pip_install target name.
+        pip_deps_repository_name: deprecated - the old {bzl:obj}`pip_parse` target name.
         manifest: the Gazelle manifest file.
             defaults to the same value as manifest.
         **kwargs: other bazel attributes passed to the generate and test targets

gazelle/manifest/defs.bzl

@@ -55,7 +55,7 @@ func main() {
 		&pipRepositoryName,
 		"pip-repository-name",
 		"",
-		"The name of the pip_install or pip_repository target.")
+		"The name of the pip_parse or pip.parse target.")
 	flag.StringVar(
 		&modulesMappingPath,
 		"modules-mapping",

gazelle/manifest/generate/generate.go

@@ -43,7 +43,6 @@ def rules_python_internal_setup():
         python_versions = sorted(TOOL_VERSIONS.keys()),
     )
 
-    # Because we don't use the pip_install rule, we have to call this to fetch its deps
     pypi_deps()
 
     bazel_skylib_workspace()

internal_setup.bzl

@@ -12,7 +12,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"pip module extension for use with bzlmod"
+"""
+This is the successor to {bzl:obj}`pip_parse` for including third party PyPI dependencies into your bazel module using `bzlmod`.
+
+:::{seealso}
+For user documentation see the [PyPI dependencies section](pypi-dependencies).
+:::
+"""
 
 load("//python/private/pypi:pip.bzl", _pip = "pip")
 

python/extensions/pip.bzl

@@ -17,6 +17,10 @@ This contains a set of rules that are used to support inclusion of third-party
 dependencies via fully locked `requirements.txt` files. Some of the exported
 symbols should not be used and they are either undocumented here or marked as
 for internal use only.
+
+If you are using a bazel version 7 or above with `bzlmod`, you should only care
+about the {bzl:obj}`compile_pip_requirements` macro exposed in this file. The
+rest of the symbols are for legacy `WORKSPACE` setups.
 """
 
 load("//python/private:normalize_name.bzl", "normalize_name")

python/pip.bzl

@@ -764,8 +764,7 @@ the BUILD files for wheels.
             attrs = _pip_parse_ext_attrs(),
             doc = """\
 This tag class is used to create a pip hub and all of the spokes that are part of that hub.
-This tag class reuses most of the pip attributes that are found in
-@rules_python//python/pip_install:pip_repository.bzl.
+This tag class reuses most of the attributes found in {bzl:obj}`pip_parse`.
 The exception is it does not use the arg 'repo_prefix'.  We set the repository
 prefix for the user and the alias arg is always True in bzlmod.
 """,
@@ -814,8 +813,7 @@ the BUILD files for wheels.
             ),
             doc = """\
 This tag class is used to create a pypi hub and all of the spokes that are part of that hub.
-This tag class reuses most of the pypi attributes that are found in
-@rules_python//python/pip_install:pip_repository.bzl.
+This tag class reuses most of the attributes found in {bzl:obj}`pip_parse`.
 The exception is it does not use the arg 'repo_prefix'.  We set the repository
 prefix for the user and the alias arg is always True in bzlmod.
 """,