feat(package): add unstable `--message-format` flag #15311

weihanglo · 2025-03-14T03:29:01Z

What does this PR try to resolve?

This adds an unstable --message-format flag to cargo package to help --list output in newline-delimited JSON format.

See https://github.com/weihanglo/cargo/blob/package-list-fmt/src/doc/man/cargo-package.md#package-options for more on what is provided.

Open questions

--list json or --message-format json, see feat(package): add unstable --message-format flag #15311 (comment)
- a single json blob vs N? What is N? See feat(package): add unstable --message-format flag #15311 (comment)
Is the current format plain or human, see feat(package): add unstable --message-format flag #15311 (comment)
snake_case or kebab-case, see feat(package): add unstable --message-format flag #15311 (comment)

How should we test and review this PR?

This currently outputs absolute paths.
If we don't want to show absolute paths in the JSON output,
could switch to relative path to either package root or cwd
(I prefer the latter though).
The actual schema, format option names, and field names is open to discuss and change. See also Determine story around JSON message compatibility #12377 for reference.

rustbot · 2025-03-14T03:29:06Z

r? @epage

rustbot has assigned @epage.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

epage · 2025-03-14T13:58:27Z

src/cargo/ops/cargo_package/mod.rs

    /// Generates a file.
    Generated(GeneratedFile),
 }

+impl serde::ser::Serialize for FileContents {


Can we have the schema defined in cargo-util-schemas with a schema.json file?

Doesn't seem like a blocker. I would like to defer until stabilization, or any request from users.

This is a schema we are expecting people to use programmatically but

We are keeping internal the helpers for doing so, leaving them to the documentation.

The serialization schema is tied directly to the implementation in a location that doesn't call out this is for serialization. This makes it harder for us to track schema changes (e.g. a refactor to rename a field). At least the impl Serialize is hand written which helps call it out but it might not always be that way. We do have tests but those can't exhaustively cover every case.

I also would like to see every existing serialization schema moved there.

I'd like to flip the script and instead of waiting on a need, for us to proactively do it. It will reduce the work we need to do later to take care of this if we do it in the first place. I held off on this for sboms mostly because I brought it up late in the process.

Yes, this could be done at stabilization. We'd need to track that as work to do for that. I also like to see stabilization make as small of a change as possible so its easier to review and has less risk.

Maybe waiting until later in the pre-stabilization process can help if there is expected churn in the overall design that it would require major rework (e.g. --list json vs --message-format json can end up with very different needs)

Extracted. Separating internal implementation makes a lot of sense!

According to our JSON compatibility policy, the schema should have room to add more fields, the current path and kind won't move away I feel like, so it should be fine doing it now than later.

There isn't a schema.json file.

However,

Its unclear what that file should look like for messages (should we put all messages in it?)

Those are most for people code-genning off of it (less needed with cargo-util-schemas) or editors (less needed for programmatic files)

So not going to block on that.

epage · 2025-03-14T14:00:03Z

src/bin/cargo/commands/package.rs

        .arg(
-            flag(
+            optional_opt(
                "list",
-                "Print files included in a package without making one",
+                "Print files included in a package without making one: (unstable) json",
            )
-            .short('l'),
+            .short('l')
+            .value_name("FMT"),


Should we instead use --message-format?

One reason is I wonder if we should focus on ndjson messages rather than a single json blob. This makes it easier to extend with more feature. For example, we can't extend cargo metadata with warnings or errors without adding them to that single schema which feels weird.

That might work, though conflating its meaning other commands with this might be a bit confusing. I personally may assume it is for diagnostics, for example missing metadata during packaging. The output of cargo package --list is not a diagnostic message.

I've taken message to mean "output message" which is generic, not specific to the type of message. The shape would probably need to be consistent across the different messages though.

Just realized that with --message-format=json the JSON messages are printed to stdout, so we are not really conflating these two stuffs. The existing compiler message are also emitted as ndjson via stdout, so reasonable 👍🏾.

There is the other question for using --message-format: Should it also affect the package verification build? How Cargo verfies packages is kinda an implementation detail, so my answer is no. However we've mentioned in doc Cargo verifies it complies. Not sure if people get confused when seeing this.

In the future --message-format=json would also affect status message Cargo emits, right? So here comes another counterpoint: This adds some more work on users side, if they want to keep the human-readable status messages.

In the future --message-format=json would also affect status message Cargo emits, right? So here comes another counterpoint: This adds some more work on users side, if they want to keep the human-readable status messages.

Thats true when interacting with most of our json APIs and why some have asked for a way to tee.

src/doc/src/reference/unstable.md

epage · 2025-03-14T14:02:13Z

src/doc/man/cargo-package.md

+  {
+    /* The Package ID Spec of the package. */
+    "path+file:///home/foo#0.0.0": {


Like I said at https://github.com/rust-lang/cargo/pull/15311/files#r1995637878, should we do a message per package or a message per file?

I am good with per package.

How would message per file look like? Tools may need more work to associate all of them, but tools usually want information for the entire package I assume?
Also, with message-per-file, Cargo then needs to guarantee that there is no interleaving in between. Otherwise tools either need to consume all messages to determine if everything in a single package is collected.

epage · 2025-03-14T14:02:28Z

src/doc/man/cargo-package.md

+      /* Relative path in the archive file. */
+      "Cargo.toml.orig": {
+        /* An absolute path to the actual file content. */
+        "original_path": "/home/foo/Cargo.toml",


snake case or kebab case?

We do that i -Zsbom, cargo metadata, and --message-format=json. This follows.

sbom is just because of cargo metadata. We deferred what it should actually do to the RFC. (I brought this up there).

Uck, didn't know --message-format=json does that.

My hope for the new plumbing commands is that we reuse public schemas as much as possible, like TomlManifest. Unless we copy/paste it, that will be kebab case.

So for right now, our schema document gives an unqualified statement to use kebab case. We should probably discuss this and figure out if there needs to be more qualifiers to it.

src/doc/man/cargo-package.md

src/cargo/ops/cargo_package/mod.rs

The behavior is not implemented yet

This is is for `cargo package --list` in JSON format

This is needed for `cargo package --list` JSON message to access the orignal file path information for the generated file kind

epage

Thanks! Looking forward to using this in cargo-release!

epage · 2025-03-26T16:32:50Z

btw I added some of the decisions made during the review discussion to the PR description so we can better track them as we move towards stabilization (so the rest of the team can re-affirm or disagree)

Update cargo 10 commits in 307cbfda3119f06600e43cd38283f4a746fe1f8b..a6c604d1b8a2f2a8ff1f3ba6092f9fda42f4b7e9 2025-03-20 20:00:39 +0000 to 2025-03-26 18:11:00 +0000 - fix(package): update tracking issue for `--message-format` (rust-lang/cargo#15354) - docs(contrib): Expand the description of team meetings (rust-lang/cargo#15349) - feat(package): add unstable `--message-format` flag (rust-lang/cargo#15311) - feat(complete): Added completion for `--profile` (rust-lang/cargo#15308) - Uplift windows Cygwin DLL import libraries (rust-lang/cargo#15193) - do not pass cdylib link args to test (rust-lang/cargo#15317) - fix: revert the behavior checking lockfile's VCS status (rust-lang/cargo#15341) - Temporarily ignore cargo_test_doctest_xcompile_ignores (rust-lang/cargo#15348) - docs: fix typo in the "Shared cache" section (rust-lang/cargo#15346) - Fix some issues with future-incompat report generation (rust-lang/cargo#15345) r? ghost

rustbot assigned epage Mar 14, 2025

epage reviewed Mar 14, 2025

View reviewed changes

src/doc/src/reference/unstable.md Outdated Show resolved Hide resolved

epage reviewed Mar 14, 2025

View reviewed changes

src/doc/man/cargo-package.md Outdated Show resolved Hide resolved

epage reviewed Mar 14, 2025

View reviewed changes

src/doc/man/cargo-package.md Outdated Show resolved Hide resolved

epage reviewed Mar 14, 2025

View reviewed changes

src/doc/man/cargo-package.md Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

weihanglo force-pushed the package-list-fmt branch from be412aa to dea55c7 Compare March 25, 2025 05:31

weihanglo changed the title ~~feat(package): add --list json option for JSON output format~~ feat(package): add unstable --message-format flag Mar 25, 2025

epage reviewed Mar 25, 2025

View reviewed changes

src/cargo/ops/cargo_package/mod.rs Outdated Show resolved Hide resolved

epage reviewed Mar 25, 2025

View reviewed changes

src/cargo/ops/cargo_package/mod.rs Outdated Show resolved Hide resolved

weihanglo force-pushed the package-list-fmt branch 3 times, most recently from 03adae7 to 6a907ac Compare March 26, 2025 02:41

weihanglo added 5 commits March 25, 2025 23:02

feat(package): add unstable --message-format flag

781c7ed

The behavior is not implemented yet

docs(package): man page for cargo package --message-format

9f9ff62

docs(unstable): --message-format flag in cargo-package

02226ae

feat(schemas): add PackageList message schema

4e8014f

This is is for `cargo package --list` in JSON format

refactor(package): attach path info to GeneratedFile

d493e38

This is needed for `cargo package --list` JSON message to access the orignal file path information for the generated file kind

feat(package): JSON message format for --list

e354769

weihanglo force-pushed the package-list-fmt branch from 6a907ac to e354769 Compare March 26, 2025 03:02

epage approved these changes Mar 26, 2025

View reviewed changes

epage added this pull request to the merge queue Mar 26, 2025

Merged via the queue into rust-lang:master with commit 856f1ba Mar 26, 2025
21 checks passed

weihanglo deleted the package-list-fmt branch March 26, 2025 17:08

weihanglo mentioned this pull request Mar 26, 2025

Tracking Issue for JSON output format of cargo package --list #15353

Open

5 tasks

weihanglo linked an issue Mar 26, 2025 that may be closed by this pull request

Provide a json output format of cargo package --list #11666

Closed

This was referenced Mar 26, 2025

Add a JSON output mode to cargo package. #14262

Closed

Provide a json output format of cargo package --list #11666

Closed

Update cargo rust-lang/rust#139005

Merged

rustbot added this to the 1.87.0 milestone Mar 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(package): add unstable `--message-format` flag #15311

feat(package): add unstable `--message-format` flag #15311

weihanglo commented Mar 14, 2025 •

edited by epage

Loading

rustbot commented Mar 14, 2025

epage Mar 14, 2025

weihanglo Mar 25, 2025

epage Mar 25, 2025

weihanglo Mar 26, 2025

epage Mar 26, 2025

epage Mar 14, 2025

weihanglo Mar 14, 2025

epage Mar 14, 2025

weihanglo Mar 14, 2025

weihanglo Mar 15, 2025

epage Mar 17, 2025

epage Mar 14, 2025

weihanglo Mar 14, 2025

epage Mar 14, 2025

weihanglo Mar 14, 2025

epage Mar 14, 2025

This comment has been minimized.

epage left a comment

epage commented Mar 26, 2025

feat(package): add unstable --message-format flag #15311

feat(package): add unstable --message-format flag #15311

Conversation

weihanglo commented Mar 14, 2025 • edited by epage Loading

What does this PR try to resolve?

How should we test and review this PR?

rustbot commented Mar 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

epage left a comment

Choose a reason for hiding this comment

epage commented Mar 26, 2025

feat(package): add unstable `--message-format` flag #15311

feat(package): add unstable `--message-format` flag #15311

weihanglo commented Mar 14, 2025 •

edited by epage

Loading