Merge branch 'main' into dcreager/union-callables

* main:
  [red-knot] Use `try_call_dunder` for augmented assignment (#16717)
  [red-knot] Document current state of attribute assignment diagnostics (#16746)
  [red-knot] Case sensitive module resolver (#16521)
  [red-knot] Very minor simplification of the render tests (#16759)
  [syntax-errors] Unparenthesized assignment expressions in sets and indexes (#16404)
  ruff_db: add a new diagnostic renderer
  ruff_db: add `context` configuration
  red_knot: plumb through `DiagnosticFormat` to the CLI
  ruff_db: add concise diagnostic mode
  [syntax-errors] Star annotations before Python 3.11 (#16545)
  [syntax-errors] Star expression in index before Python 3.11 (#16544)
  Ruff 0.11.0 (#16723)
  [red-knot] Preliminary tests for typing.Final (#15917)
  [red-knot] fix: improve type inference for binary ops on tuples (#16725)
  [red-knot] Assignments to attributes (#16705)
  [`pygrep-hooks`]: Detect file-level suppressions comments without rul… (#16720)
  Fallback to requires-python in certain cases when target-version is not found (#16721)

Author: dcreager

BREAKING_CHANGES.md

@@ -1,6 +1,8 @@
 # Breaking Changes
 
-## 0.10.0
+## 0.11.0
+
+This is a follow-up to release 0.10.0. Because of a mistake in the release process, the `requires-python` inference changes were not included in that release. Ruff 0.11.0 now includes this change as well as the stabilization of the preview behavior for `PGH004`.
 
 - **Changes to how the Python version is inferred when a `target-version` is not specified** ([#16319](https://github.com/astral-sh/ruff/pull/16319))
 
@@ -23,6 +25,13 @@
         search for the closest `pyproject.toml` in the parent directories and use its
         `requires-python` setting.
 
+## 0.10.0
+
+- **Changes to how the Python version is inferred when a `target-version` is not specified** ([#16319](https://github.com/astral-sh/ruff/pull/16319))
+
+    Because of a mistake in the release process, the `requires-python` inference changes are not included in this release and instead shipped as part of 0.11.0.
+    You can find a description of this change in the 0.11.0 section.
+
 - **Updated `TYPE_CHECKING` behavior** ([#16669](https://github.com/astral-sh/ruff/pull/16669))
 
     Previously, Ruff only recognized typechecking blocks that tested the `typing.TYPE_CHECKING` symbol. Now, Ruff recognizes any local variable named `TYPE_CHECKING`. This release also removes support for the legacy `if 0:` and `if False:` typechecking checks. Use a local `TYPE_CHECKING` variable instead.

CHANGELOG.md

@@ -1,13 +1,11 @@
 # Changelog
 
-## 0.10.0
+## 0.11.0
 
-Check out the [blog post](https://astral.sh/blog/ruff-v0.10.0) for a migration guide and overview of the changes!
+This is a follow-up to release 0.10.0. Because of a mistake in the release process, the `requires-python` inference changes were not included in that release. Ruff 0.11.0 now includes this change as well as the stabilization of the preview behavior for `PGH004`.
 
 ### Breaking changes
 
-See also, the "Remapped rules" section which may result in disabled rules.
-
 - **Changes to how the Python version is inferred when a `target-version` is not specified** ([#16319](https://github.com/astral-sh/ruff/pull/16319))
 
     In previous versions of Ruff, you could specify your Python version with:
@@ -29,6 +27,29 @@ See also, the "Remapped rules" section which may result in disabled rules.
         search for the closest `pyproject.toml` in the parent directories and use its
         `requires-python` setting.
 
+### Stabilization
+
+The following behaviors have been stabilized:
+
+- [`blanket-noqa`](https://docs.astral.sh/ruff/rules/blanket-noqa/) (`PGH004`): Also detect blanked file-level noqa comments (and not just line level comments).
+
+### Preview features
+
+- [syntax-errors] Tuple unpacking in `for` statement iterator clause before Python 3.9 ([#16558](https://github.com/astral-sh/ruff/pull/16558))
+
+## 0.10.0
+
+Check out the [blog post](https://astral.sh/blog/ruff-v0.10.0) for a migration guide and overview of the changes!
+
+### Breaking changes
+
+See also, the "Remapped rules" section which may result in disabled rules.
+
+- **Changes to how the Python version is inferred when a `target-version` is not specified** ([#16319](https://github.com/astral-sh/ruff/pull/16319))
+
+    Because of a mistake in the release process, the `requires-python` inference changes are not included in this release and instead shipped as part of 0.11.0.
+    You can find a description of this change in the 0.11.0 section.
+
 - **Updated `TYPE_CHECKING` behavior** ([#16669](https://github.com/astral-sh/ruff/pull/16669))
 
     Previously, Ruff only recognized typechecking blocks that tested the `typing.TYPE_CHECKING` symbol. Now, Ruff recognizes any local variable named `TYPE_CHECKING`. This release also removes support for the legacy `if 0:` and `if False:` typechecking checks. Use a local `TYPE_CHECKING` variable instead.
@@ -86,7 +107,6 @@ The following behaviors have been stabilized:
 
 - [`bad-staticmethod-argument`](https://docs.astral.sh/ruff/rules/bad-staticmethod-argument/) (`PLW0211`) [`invalid-first-argument-name-for-class-method`](https://docs.astral.sh/ruff/rules/invalid-first-argument-name-for-class-method/) (`N804`): `__new__` methods are now no longer flagged by `invalid-first-argument-name-for-class-method` (`N804`) but instead by `bad-staticmethod-argument` (`PLW0211`)
 - [`bad-str-strip-call`](https://docs.astral.sh/ruff/rules/bad-str-strip-call/) (`PLE1310`): The rule now applies to objects which are known to have type `str` or `bytes`.
-- [`blanket-noqa`](https://docs.astral.sh/ruff/rules/blanket-noqa/) (`PGH004`): Also detect blanked file-level noqa comments (and not just line level comments).
 - [`custom-type-var-for-self`](https://docs.astral.sh/ruff/rules/custom-type-var-for-self/) (`PYI019`): More accurate detection of custom `TypeVars` replaceable by `Self`. The range of the diagnostic is now the full function header rather than just the return annotation.
 - [`invalid-argument-name`](https://docs.astral.sh/ruff/rules/invalid-argument-name/) (`N803`): Ignore argument names of functions decorated with `typing.override`
 - [`invalid-envvar-default`](https://docs.astral.sh/ruff/rules/invalid-envvar-default/) (`PLW1508`): Detect default value arguments to `os.environ.get` with invalid type.

Cargo.lock

@@ -154,6 +154,7 @@ toml = { version = "0.8.11" }
 tracing = { version = "0.1.40" }
 tracing-flame = { version = "0.2.0" }
 tracing-indicatif = { version = "0.3.6" }
+tracing-log = { version = "0.2.0" }
 tracing-subscriber = { version = "0.3.18", default-features = false, features = [
     "env-filter",
     "fmt",

Cargo.toml

@@ -149,8 +149,8 @@ curl -LsSf https://astral.sh/ruff/install.sh | sh
 powershell -c "irm https://astral.sh/ruff/install.ps1 | iex"
 
 # For a specific version.
-curl -LsSf https://astral.sh/ruff/0.10.0/install.sh | sh
-powershell -c "irm https://astral.sh/ruff/0.10.0/install.ps1 | iex"
+curl -LsSf https://astral.sh/ruff/0.11.0/install.sh | sh
+powershell -c "irm https://astral.sh/ruff/0.11.0/install.ps1 | iex"
 ```
 
 You can also install Ruff via [Homebrew](https://formulae.brew.sh/formula/ruff), [Conda](https://anaconda.org/conda-forge/ruff),
@@ -183,7 +183,7 @@ Ruff can also be used as a [pre-commit](https://pre-commit.com/) hook via [`ruff
 ```yaml
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.10.0
+  rev: v0.11.0
   hooks:
     # Run the linter.
     - id: ruff

README.md

@@ -75,6 +75,10 @@ pub(crate) struct CheckCommand {
     #[clap(flatten)]
     pub(crate) rules: RulesArg,
 
+    /// The format to use for printing diagnostic messages.
+    #[arg(long)]
+    pub(crate) output_format: Option<OutputFormat>,
+
     /// Use exit code 1 if there are any warning-level diagnostics.
     #[arg(long, conflicts_with = "exit_zero", default_missing_value = "true", num_args=0..1)]
     pub(crate) error_on_warning: Option<bool>,
@@ -117,6 +121,9 @@ impl CheckCommand {
                 ..EnvironmentOptions::default()
             }),
             terminal: Some(TerminalOptions {
+                output_format: self
+                    .output_format
+                    .map(|output_format| RangedValue::cli(output_format.into())),
                 error_on_warning: self.error_on_warning,
             }),
             rules,
@@ -211,3 +218,32 @@ impl clap::Args for RulesArg {
         Self::augment_args(cmd)
     }
 }
+
+/// The diagnostic output format.
+#[derive(Copy, Clone, Hash, Debug, PartialEq, Eq, PartialOrd, Ord, Default, clap::ValueEnum)]
+pub enum OutputFormat {
+    /// Print diagnostics verbosely, with context and helpful hints.
+    ///
+    /// Diagnostic messages may include additional context and
+    /// annotations on the input to help understand the message.
+    #[default]
+    #[value(name = "full")]
+    Full,
+    /// Print diagnostics concisely, one per line.
+    ///
+    /// This will guarantee that each diagnostic is printed on
+    /// a single line. Only the most important or primary aspects
+    /// of the diagnostic are included. Contextual information is
+    /// dropped.
+    #[value(name = "concise")]
+    Concise,
+}
+
+impl From<OutputFormat> for ruff_db::diagnostic::DiagnosticFormat {
+    fn from(format: OutputFormat) -> ruff_db::diagnostic::DiagnosticFormat {
+        match format {
+            OutputFormat::Full => Self::Full,
+            OutputFormat::Concise => Self::Concise,
+        }
+    }
+}

crates/red_knot/src/args.rs

@@ -80,6 +80,8 @@ fn run_check(args: CheckCommand) -> anyhow::Result<ExitStatus> {
     countme::enable(verbosity.is_trace());
     let _guard = setup_tracing(verbosity)?;
 
+    tracing::debug!("Version: {}", version::version());
+
     // The base path to which all CLI arguments are relative to.
     let cwd = {
         let cwd = std::env::current_dir().context("Failed to get the current working directory")?;
@@ -256,6 +258,7 @@ impl MainLoop {
                     revision: check_revision,
                 } => {
                     let display_config = DisplayDiagnosticConfig::default()
+                        .format(db.project().settings(db).terminal().output_format)
                         .color(colored::control::SHOULD_COLORIZE.should_colorize());
 
                     let min_error_severity =

crates/red_knot/src/main.rs

@@ -967,6 +967,30 @@ fn check_non_existing_path() -> anyhow::Result<()> {
     Ok(())
 }
 
+#[test]
+fn concise_diagnostics() -> anyhow::Result<()> {
+    let case = TestCase::with_file(
+        "test.py",
+        r#"
+        print(x)     # [unresolved-reference]
+        print(4[1])  # [non-subscriptable]
+        "#,
+    )?;
+
+    assert_cmd_snapshot!(case.command().arg("--output-format=concise"), @r"
+    success: false
+    exit_code: 1
+    ----- stdout -----
+    warning[lint:unresolved-reference] <temp_dir>/test.py:2:7: Name `x` used when not defined
+    error[lint:non-subscriptable] <temp_dir>/test.py:3:7: Cannot subscript object of type `Literal[4]` with no `__getitem__` method
+    Found 2 diagnostics
+
+    ----- stderr -----
+    ");
+
+    Ok(())
+}
+
 struct TestCase {
     _temp_dir: TempDir,
     _settings_scope: SettingsBindDropGuard,

crates/red_knot/tests/cli.rs

@@ -1,5 +1,3 @@
-#![allow(clippy::disallowed_names)]
-
 use std::collections::HashSet;
 use std::io::Write;
 use std::time::{Duration, Instant};
@@ -16,7 +14,7 @@ use ruff_db::source::source_text;
 use ruff_db::system::{
     OsSystem, System, SystemPath, SystemPathBuf, UserConfigDirectoryOverrideGuard,
 };
-use ruff_db::Upcast;
+use ruff_db::{Db as _, Upcast};
 use ruff_python_ast::PythonVersion;
 
 struct TestCase {
@@ -1790,3 +1788,82 @@ fn changes_to_user_configuration() -> anyhow::Result<()> {
 
     Ok(())
 }
+
+/// Tests that renaming a file from `lib.py` to `Lib.py` is correctly reflected.
+///
+/// This test currently fails on case-insensitive systems because `Files` is case-sensitive
+/// but the `System::metadata` call isn't. This means that
+/// Red Knot considers both `Lib.py` and `lib.py` to exist when only `lib.py` does
+///
+/// The incoming change events then are no-ops because they don't change either file's
+/// status nor does it update their last modified time (renaming a file doesn't bump it's
+/// last modified timestamp).
+///
+/// Fixing this requires to either make `Files` case-insensitive and store the
+/// real-case path (if it differs) on `File` or make `Files` use a
+/// case-sensitive `System::metadata` call. This does open the question if all
+/// `System` calls should be case sensitive. This would be the most consistent
+/// but might be hard to pull off.
+///
+/// What the right solution is also depends on if Red Knot itself should be case
+/// sensitive or not. E.g. should `include="src"` be case sensitive on all systems
+/// or only on case-sensitive systems?
+///
+/// Lastly, whatever solution we pick must also work well with VS Code which,
+/// unfortunately ,doesn't propagate casing-only renames.
+/// <https://github.com/rust-lang/rust-analyzer/issues/9581>
+#[ignore]
+#[test]
+fn rename_files_casing_only() -> anyhow::Result<()> {
+    let mut case = setup([("lib.py", "class Foo: ...")])?;
+
+    assert!(
+        resolve_module(case.db(), &ModuleName::new("lib").unwrap()).is_some(),
+        "Expected `lib` module to exist."
+    );
+    assert_eq!(
+        resolve_module(case.db(), &ModuleName::new("Lib").unwrap()),
+        None,
+        "Expected `Lib` module not to exist"
+    );
+
+    // Now rename `lib.py` to `Lib.py`
+    if case.db().system().case_sensitivity().is_case_sensitive() {
+        std::fs::rename(
+            case.project_path("lib.py").as_std_path(),
+            case.project_path("Lib.py").as_std_path(),
+        )
+        .context("Failed to rename `lib.py` to `Lib.py`")?;
+    } else {
+        // On case-insensitive file systems, renaming a file to a different casing is a no-op.
+        // Rename to a different name first
+        std::fs::rename(
+            case.project_path("lib.py").as_std_path(),
+            case.project_path("temp.py").as_std_path(),
+        )
+        .context("Failed to rename `lib.py` to `temp.py`")?;
+
+        std::fs::rename(
+            case.project_path("temp.py").as_std_path(),
+            case.project_path("Lib.py").as_std_path(),
+        )
+        .context("Failed to rename `temp.py` to `Lib.py`")?;
+    }
+
+    let changes = case.stop_watch(event_for_file("Lib.py"));
+    case.apply_changes(changes);
+
+    // Resolving `lib` should now fail but `Lib` should now succeed
+    assert_eq!(
+        resolve_module(case.db(), &ModuleName::new("lib").unwrap()),
+        None,
+        "Expected `lib` module to no longer exist."
+    );
+
+    assert!(
+        resolve_module(case.db(), &ModuleName::new("Lib").unwrap()).is_some(),
+        "Expected `Lib` module to exist"
+    );
+
+    Ok(())
+}

crates/red_knot/tests/file_watching.rs

@@ -2,7 +2,7 @@ use crate::metadata::value::{RangedValue, RelativePathBuf, ValueSource, ValueSou
 use crate::Db;
 use red_knot_python_semantic::lint::{GetLintError, Level, LintSource, RuleSelection};
 use red_knot_python_semantic::{ProgramSettings, PythonPath, PythonPlatform, SearchPathSettings};
-use ruff_db::diagnostic::{DiagnosticId, OldDiagnosticTrait, Severity, Span};
+use ruff_db::diagnostic::{DiagnosticFormat, DiagnosticId, OldDiagnosticTrait, Severity, Span};
 use ruff_db::files::system_path_to_file;
 use ruff_db::system::{System, SystemPath};
 use ruff_macros::Combine;
@@ -120,6 +120,11 @@ impl Options {
 
         if let Some(terminal) = self.terminal.as_ref() {
             settings.set_terminal(TerminalSettings {
+                output_format: terminal
+                    .output_format
+                    .as_deref()
+                    .copied()
+                    .unwrap_or_default(),
                 error_on_warning: terminal.error_on_warning.unwrap_or_default(),
             });
         }
@@ -277,6 +282,11 @@ impl FromIterator<(RangedValue<String>, RangedValue<Level>)> for Rules {
 #[serde(rename_all = "kebab-case", deny_unknown_fields)]
 #[cfg_attr(feature = "schemars", derive(schemars::JsonSchema))]
 pub struct TerminalOptions {
+    /// The format to use for printing diagnostic messages.
+    ///
+    /// Defaults to `full`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub output_format: Option<RangedValue<DiagnosticFormat>>,
     /// Use exit code 1 if there are any warning-level diagnostics.
     ///
     /// Defaults to `false`.

crates/red_knot_project/src/metadata/options.rs

@@ -1,6 +1,7 @@
 use std::sync::Arc;
 
 use red_knot_python_semantic::lint::RuleSelection;
+use ruff_db::diagnostic::DiagnosticFormat;
 
 /// The resolved [`super::Options`] for the project.
 ///
@@ -49,5 +50,6 @@ impl Settings {
 
 #[derive(Debug, Clone, PartialEq, Eq, Default)]
 pub struct TerminalSettings {
+    pub output_format: DiagnosticFormat,
     pub error_on_warning: bool,
 }