pkg_resources.resource_string allows absolute paths and paths with .. - contrary to docs

[Mekk]

The https://setuptools.readthedocs.io/en/latest/pkg_resources.html ("Basic Resource Access") page claims:

Note that resource names must be /-separated paths and cannot be absolute (i.e. no leading /)
or contain relative names like "..".

Let's see:

>>> pkg_resources.resource_string('multiprocessing', '/__init__.py')
'#\n# Package analogous …

>>> pkg_resources.resource_string('multiprocessing', '../../../../etc/passwd')
'root:x:0:0:root…

I'd say some validation is missing.

Tested on both python2.7 and python3.6, with pkg_resources as in Ubuntu 18.04

[Mekk]

Note that this may have security impact… I already saw the code, which assumed that using pkg_resources guarantees that given path belongs to the package (so can be shown/safely used/…).

[jaraco]

I've started working on this in https://github.com/pypa/setuptools/tree/bugfix/1635-disallow-parent-paths but with that change, I notice that there are many DeprecationWarnings in the setuptools codebase itself, including this call:

setuptools/pkg_resources/__init__.py

Line 1936 in 82ae7bb

for subitem in metadata.resource_listdir('/'):

Due to the way the value is used, by splitting on / and passing the result to os.path.join, the leading / has no effect:

>>> '/'.split('/')
['', '']
>>> os.path.join('foo', '', '')
'foo/'
>>> os.path.join('foo', '', '', '')
'foo/'

So in the case of '/', I suggest just changing the docs to update the expectation.

[jaraco]

@Mekk Would you review the PR and share your thoughts?

[Mekk]

1. In my opinion leading / should be forbidden. It simply adds alternative confusing syntax. And there are various subtle implications.

   To give you a taste:

   • there exist real code which translates between url-s and pkg_resource apis, for example add_static_view in Pyramid. And in the context of url translation such extra slash may have non-trivial meaning…

   • such a slash may have meaning in case of non-filesystem-based resources, like various archives (some rar or another zip may be happy to extract some/file but not necessarily /some/file). Making restrictions consistent avoid such issues (this is my real case behind this bug, people used /slash/leading/paths, this worked for them on dev, then on true archive-based installation it failed)

   In general: pkg_resources should work the same whether package is on filesystem, in egg, or even provided by some custom loader (as long as that one fulfills specs).. Having some paths working on fs-based installation but not working elsewhere is painful.

2. Regarding actual patch: what's going to happen on Windows, if \..\ happens (backslash-based illegal path)?

3. Regarding docs: I'd add some note that currently invalid paths cause only warnings in case of filesystem-installed packages, but this may change into error in the future, and may fail with custom loaders even just now.

[jaraco]

The latest patch also disallows the leading '/'. I imagine there are hundreds of cases that will be affected by this change, so a longer deprecation period will be required.

what's going to happen on Windows?

Possibly a similar issue. There was no protection for this issue before.

I'd add some note that currently invalid paths cause only warnings

I'd like to rely on the changelog to document transient aspects and allow the documentation to reflect the more permanent intentions/expectations.

[jaraco]

what's going to happen on Windows?

On Windows, if a \ or C:\ is present at the beginning, it will give access to the whole system:

>>> ntpath.join('foo', 'C:\\bar.txt')                                                                                                         
'C:\\bar.txt'

So I've updated the PR to unconditionally disallow Windows-based absolute paths.

[airvzxf]

Hi, I am disagreed to this change because it is not preventing the security since I am able to use setup.py and take any file from the computer, or I am able to modify manually the system file setuptools/config.py and remove the lines of code, I mean it is not a binary compiled and striped. With all of this, I try to say that the philosophy of Python is that it create a freedom kingdom for developers because all of us are adults.

Second, this change is blocking to me and others, since I searched on Internet about this problem. Why setuptools/setup.cfg is forcing me to have all the files in one directory, I understand that the structure having all files in the root is very common (same problem with Dockerfile), but at the same time is nasty, here is my tree directory. In this real example the code is in the main folder, the read me file is in ./README.md and the source code is in ./src/ which include the PyPi setup files.

.
├── LICENSE
├── README.md
├── requirements.txt
├── src
│   ├── build
│   │   ├── bdist.linux-x86_64
│   │   └── lib
│   │       └── sniparinject
│   │           ├── core
│   │           │   ├── game.py
│   │           │   ├── __init__.py
│   │           │   ├── settings.py
│   │           │   ├── text_style.py
│   │           │   └── utility.py
│   │           ├── __init__.py
│   │           └── network_sniffer.py
│   ├── dist
│   │   ├── sniparinject-0.0.0.dev5-py3-none-any.whl
│   │   └── sniparinject-0.0.0.dev5.tar.gz
│   ├── __init__.py
│   ├── pyproject.toml
│   ├── setup.cfg
│   ├── sniparinject
│   │   ├── core
│   │   │   ├── game.py
│   │   │   ├── __init__.py
│   │   │   ├── settings.py
│   │   │   ├── text_style.py
│   │   │   └── utility.py
│   │   ├── __init__.py
│   │   ├── network_sniffer.py
│   └── sniparinject.egg-info
│       ├── dependency_links.txt
│       ├── PKG-INFO
│       ├── requires.txt
│       ├── SOURCES.txt
│       └── top_level.txt
├── test
│   ├── __init__.py
│   ├── pytest.ini
│   ├── test_game.py
│   ├── test_network_sniffer.py
│   ├── test_settings.py
│   └── test_utility.py
└── TODO.md

The setup.cfg file contains:

[metadata]
; .... more set up here
long_description = file: ../README.md
long_description_content_type = text/markdown
url = https://github.com/airvzxf/sniparinject
; .... more set up here

---

As, I mentioned above, it is easy to fix with setup.py but I want to use setup.cfg because I like it. Means here the security is not covered since I can do it in one place and not in the other.

import setuptools

with open('../README.md', 'r', encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    long_description=long_description,
)

---

Other important reason is that it is using only to read files and put the content inside the PKG-INFO, which copy the content of the README.md file into the file mentioned. I understand the part of copy files from root, but copy a text file to the PKG-INFO?

In the file setuptools/config.py:

    @classmethod
    def _parse_file(cls, value):
        """Represents value as a string, allowing including text
        from nearest files using `file:` directive.

        Directive is sandboxed and won't reach anything outside
        directory with setup.py.

        Examples:
            file: README.rst, CHANGELOG.md, src/file.txt

        :param str value:
        :rtype: str
        """
        include_directive = 'file:'

        if not isinstance(value, str):
            return value

        if not value.startswith(include_directive):
            return value

        spec = value[len(include_directive):]
        filepaths = (os.path.abspath(path.strip()) for path in spec.split(','))
        return '\n'.join(
            cls._read_file(path)
            for path in filepaths
            if (cls._assert_local(path) or True)
            and os.path.isfile(path)
        )

    @staticmethod
    def _assert_local(filepath):
        if not filepath.startswith(os.getcwd()):
            raise DistutilsOptionError(
                '`file:` directive can not access %s' % filepath)

    @staticmethod
    def _read_file(filepath):
        with io.open(filepath, encoding='utf-8') as f:
            return f.read()

More information in the issue that I created: #2699