Black stops looking for config at first pyproject.toml file even if there's no black section in it

[slafs]

Disclaimer: I was hesitant whether I should report this as a bug or rather a "Feature request".

Describe the bug

In a project dealing with multiple sub-projects in a single repository (aka. mono-repo style) having more than one pyproject.toml file confuses black.
In our example we wanted to have one top-level black configuration,
while also having sub-project specific pyproject.toml files (without any black config) in each sub-directory.
Unfortunately, running black on a sub-directory
uses default configuration instead of the one defined in the top-level config file.

To Reproduce

Given this file structure:

.
|-- pyproject.toml
|-- sub1
|   |-- pyproject.toml
|   `-- test.py
`-- sub2
    `-- test.py

With the following contents:

==> ./pyproject.toml <==
[tool.black]
line-length = 20

==> ./sub1/pyproject.toml <==
[tool.isort]
profile = "black"

==> ./sub1/test.py <==
a_very_long_line = {"that should be wrapped": "to fit 20 characters line length"}

==> ./sub2/test.py <==
a_very_long_line = {"that should be wrapped": "to fit 20 characters line length"}

Running black on both test.py files:

$ black -v sub1/test.py
Identified `/private/tmp/black-many-configs/sub1` as project root containing a pyproject.toml.
Sources to be formatted: "test.py"
sub1/test.py already well formatted, good job.

All done! ✨ 🍰 ✨
1 file left unchanged.

$ black -v sub2/test.py
Identified `/private/tmp/black-many-configs` as project root containing a pyproject.toml.
Sources to be formatted: "sub2/test.py"
Using configuration from project root.
reformatted sub2/test.py

All done! ✨ 🍰 ✨
1 file reformatted.

Modifies only the file inside sub2 directory (the one without pyproject.toml file)
even though sub1/pyproject.toml doesn't contain any black config section.

Expected behavior

Both files should be modified
and black should continue looking for a file
that is relevant for its configuration (in this case, the one that has a [tool.black] section).

Environment

• Black's version: black, 22.1.0 (compiled: yes)
• OS and Python version: OSX - Python 3.9.9

Additional context

A variation of this problem has already been reported in #1826, where the proposed solution is to introduce support for the config option inside the sub-project's pyproject.toml config file.

Other (potentially) relevant issues: #2537, #2850.

[felix-hilden]

Hi! This is indeed intended behavior. I'm not sure if we should go beyond if we don't find a Black section, but I'm not totally against it either. But would #1826 be enough for your use case? It has an active PR.

[slafs]

But would #1826 be enough for your use case?

Thanks. I think it would, yes.
Although IMHO this behaviour is confusing, given that pyproject.toml is becoming (or even already is?) the de-facto standard for tooling configuration.

[felix-hilden]

Although IMHO this behaviour is confusing

Could you elaborate a bit? The way I see it, #1826 would make the association explicit. If you wanted to leave one of your projects' configuration empty, you can now leave it empty, but you would have to have a tool.black with nothing in it. That's fine, but forgetting to put it could be confusing. Honestly, not sure, but I like the explicitness and lack of guessing what the intent is.

[Shivansh-007]

Hey 👋🏻,

Firstly, according to me none of your linked relevant issues is really relevant to this, so that could be edited way. Secondly, yeah for now #2525 (PR for 1826) should work by adding config = "../pyproject.toml to sub1/pyproject.toml.

This is also an issue when you are working with mono repo, say you have black as a repository with blib2to3 as separate repositories within black. Now, if you are in blib2to3 and the black config is outside in the root directory black, it would never go beyond blib2to3 as it exists after finding a .git folder inside it.

A possible solution to this would be to go through all the common_base.parents and find the ones which fulfil the condition of .git/.hg/pyproject.toml. Now you check for pyproject.toml in them and see if they are valid, and you return the first path which has valid pyproject.toml. Okay, this is a bit confusing ig, so say if you have:

/, /home/, /home/shivansh/, /home/shivansh/oss/, /home/shivansh/oss/black, /home/shivansh/oss/black/blib2to3

as the common_base.parents, now if you check for those which fulfil the conditions, and you get

/home/shivansh/oss/, /home/shivansh/oss/black, /home/shivansh/oss/black/blib2to3

you check for valid pyproject.toml and get oss/ and oss/black have valid configs, but since oss/black is "higher" you return that. If none of them is valid, you return the as-it-is root which would be blib2to3, if you have no path fulfilling the condition, the root is / as we have now.

Note: This is only for finding the config, the src root would stay the same as now.

Well yeah, this has the same problem as Felix mentioned, where you would need to have an empty black config in case you want to have an empty config.

---

The second solution, I can think of is to have no special case for this and just make the users link config= (#2525) to the "actual" root config. I am slightly leaning towards this for simplicity.

[cooperlees]

I feel we should only find the first pyproject.toml in the hierarchy and just error that there is no black config in the file we find. Then the user can fix their repo. I don't think black should do anything clever here. I'm a big KISS (Keep it simple stupid) person. The second solution to allow specifying the config sounds a good middle ground.

Am I missing something here?

E.g. .flake8 does this, but is more explicit as it uses a non shared config file, but if an an empty / incomplete config file exists it will cause unexpected behaviors ... I'm not saying this is ideal just comparing to other tools in the ecosystem

[slafs]

Could you elaborate a bit?

Oh, sorry @felix-hilden, I didn't mean the behaviour/solution in #1826. It would improve the current situation and it is explicit (even if slightly more verbose).

What I meant was that the assumption that black config is and should be in the first encountered pyproject.toml might be surprising to the user. In this vain raising an error as @cooperlees is proposing is also surprising, at least to me. I mean, wouldn't it be surprising to you when you have your tool X config in exactly one place in the same directory you're invoking that tool from and it doesn't get picked up?

I guess (and I might be entirely off here, so correct me if needed or ignore this paragraph) the problem origins from the facts that (a) mono-repo layouts weren't considered when black was being conceived and (b) back in the day black was one of the few tools that was using pyproject.toml file for config, so I guess its assumption was "oh, I see this file, no one else probably cares about it, so my config must be in there". Yes, I've just anthropomorphised black 😄.

On the other hand, it's hard to find consistency in other tools for this behaviour. I personally like the way isort discovers config. Mypy for example doesn't even go beyond current working directory when searching for config. So... 🤷

[Shivansh-007]

Just looked at how isort handles config, and I really like the whole config component. It's somewhat similar to what I proposed above in the first solution, not sure how it handles the empty config problem though

Well yeah, this has the same problem as Felix mentioned, where you would need to have an empty black config in case you want to have an empty config.

[blackliner]

Looks like Where Black looks for the file does not make this special situation (multiple configs) better:

By default Black looks for pyproject.toml starting from the common base directory of all files and directories passed on the command line.

Had that issue just now. I would propose to

• only search for a config starting from each file independently. This way it is always clear which file is chosen, independently how black is called (one or multiple files)
• skip configs that do not contain a black tool section until one is found (in / 🙈 )

[Shivansh-007]

This issue seems related to #2598 (just came across it)

[jaklan]

I also find it pretty problematic in a monorepo setup:

• pre-commit hook doesn't work as you can expect unless you provide --config flag explicitly, which can be really confusing (it took me a while to understand why files are formatted with no respect to the config), but when you specify it eventually - it works fine
• VS Code is more problematic - here you also have to specify Python › Formatting: Black Args explicitly, but that's more tricky because there's no obvious way to share the config with your team - workspace-level settings.json is not universal, so you need to create sth like settings.json.default and tell people to include manually / with some custom script the settings specified there in their own config. That's ofc doable, but it's a pretty annoying experience for new-comers.

If Black would scan all the pyproject.toml files up to the root of a monorepo - nothing from the above would be needed. And empty Black section in pyproject.toml as a way to say "don't scan further, just use Black defaults" should provide pretty smooth experience.

EDIT: I just had a look at #2525 - that should also be fine, I can handle specifying the path during project generation with cookiecutter then. It would be nice to be able to specify the path also from the project root, not only relatively, but I understand that would complicate the setup. @Shivansh-007 - is that PR expected to be merged anytime soon?

[jaklan]

@Shivansh-007 I did a quick test with isort - if there's intermediate pyproject.toml, but there's no isort section there - it searches further and finally pick up the config from the root, so that's exactly the behaviour we would like to see also in Black.

[Shivansh-007]

Yeah, that PR is expected to merge sometime soon, it is waiting for @ichard26's review, which would probably be small changes if not an approval.

Yeah thanks for looking into that, so it's similar to what I proposed originally.

[jaklan]

@Shivansh-007 any updates? We have already centralised almost all the configs in our monorepo, but black is still a problem 😕

[Shivansh-007]

I have been super busy for the past month or two due to school, so I haven't been able to respond to any of the open projects, so no updates as of now.

I just had a look at the PR and looks like it's still waiting for @ichard26's review.

[rushilsrivastava]

It doesn't seem like #2525 is going to be merged soon. I'm curious to hear how others are working around this

[daniel-burkhardt-keyrock-eu]

Ruff does this https://docs.astral.sh/ruff/configuration/#config-file-discovery