Requirement already satisfied by egg but package not installed #3710

jaraco · 2016-05-21T15:27:58Z

Pip version: 8.1.2
Python version: 3.5.1
Operating System: OS X 10.11.5

Description:

In this example, I'm using setuptools because that's the package I was developing, but the issue could be replicated with any package.

I'd just finished developing setuptools and wanted to install a final version, so I did python setup.py develop --uninstall. Now I have no setuptools installed, so I want to use pip to install it, but when I attempt to do so, it fails to install, saying the requirement is already satisfied. However, the egg it references about being already satisfied isn't actually on sys.path, so it's not installed (and thus not satisfied for install).

It seems that the presence of an egg from a previous install of setuptools is blocking the installation of any version.

As I'm working toward getting away from using easy_install and relying more on pip, I wanted to use pip to install setuptools rather than use the bootstrap script or install from source.

I would expect pip to be able to install the requested in this case, even if it has to uninstall extant eggs or just add them to the easy-install.pth file to make them installed.

I have found that if I repeatedly run pip unistall -y setuptools until it no longer finds any versions, I can then install setuptools.

What I've run:

$ python -m pip install setuptools
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/setuptools-21.0.0-py3.5.egg
$ python -c "import setuptools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'setuptools'
$ pip freeze | grep setuptools
$ pip uninstall -y setuptools
...
$ pip uninstall -y setuptools
...
$ pip uninstall -y setuptools
...
$ python -m pip uninstall -y setuptools
Cannot uninstall requirement setuptools, not installed
$ python -m pip install --egg setuptools
Collecting setuptools
  Using cached setuptools-21.1.0-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-21.1.0

The text was updated successfully, but these errors were encountered:

dstufft · 2016-09-07T20:45:47Z

This is probably a setuptools bug? I think we're just using pkg_resources to iterate over the installed distributions. Unless we're using pkg_resources wrong?

jaraco · 2016-09-07T21:11:36Z

Well, the packages aren't "installed" in the sense that they can't be imported without first being "required" by pkg_resources (they're not listed in easy-install.pth). They are "installed" in the hidden multi-package sense.

Do you know which call to pkg_resources is used to determine what's installed? Does it make sense for pip to consider hidden packages as installed and satisfying its install operation?

dstufft · 2016-09-07T21:15:25Z

It probably does not make sense for hidden packages to count as installed.

You can see the call to this at pip/req/req_install.py#L992-L1031, but the call we're using is:

self.satisfied_by = pkg_resources.get_distribution("REQNAME")

jaraco · 2016-09-07T21:19:58Z

I'll definitely take a look at this in more depth later. Thanks for the reference.

jaraco · 2016-09-16T21:33:56Z

So I've traced the call to get_distribution to this call in get_provider, in particular:

require(str(moduleOrReq))[0]

So it seems that intentional or not, get_distribution will return a distribution not in the working set but one that's reachable through require (which could be an old egg install).

I'm thinking there are two ways to deal with this:

setuptools should expose and pip should consume a new interface that only returns distributions in the working set (not old eggs or multi-installed eggs).
pip should uninstall every possible distribution and not just the first one. That is, it should repeatedly invoke uninstall until pkg_resources.get_distribution doesn't return anything.

I'm leaning toward the first option, meaning that pip wouldn't touch distributions that would not be importable (since it also won't install distributions that won't be importable).

Thoughts?

dstufft · 2016-09-16T22:01:34Z

The second option will probably not mesh well with Debians patching. They block us from uninstalling stuff that is installed by the OS. The first option sounds like a good idea to me.

Sent from my iPhone

On Sep 16, 2016, at 5:33 PM, Jason R. Coombs [email protected] wrote:

So I've traced the call to get_distribution to this call in get_provider, in particular:

require(str(moduleOrReq))[0]
So it seems that intentional or not, get_distribution will return a distribution not in the working set but one that's reachable through require (which could be an old egg install).

I'm thinking there are two ways to deal with this:

setuptools should expose and pip should consume a new interface that only returns distributions in the working set (not old eggs or multi-installed eggs).
pip should uninstall every possible distribution and not just the first one. That is, it should repeatedly invoke uninstall until pkg_resources.get_distribution doesn't return anything.
I'm leaning toward the first option, meaning that pip wouldn't touch distributions that would not be importable (since it also won't install distributions that won't be importable).

Thoughts?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

xavfernandez · 2016-10-14T21:28:16Z

First option also 👍

nishnik · 2016-11-13T01:17:53Z

For people struggling with hack to get it right, I had similar issue:

nishnik@nishnik-Lenovo-Y50-70:~$ pip uninstall selenium
Cannot uninstall requirement selenium, not installed
nishnik@nishnik-Lenovo-Y50-70:~$ sudo pip3 install selenium --user
Requirement already satisfied (use --upgrade to upgrade): selenium in /usr/local/lib/python3.5/dist-packages

I deleted the directory /usr/local/lib/python3.5/dist-packages/selenium and then pip installed it again, it worked right for me.

pradyunsg · 2018-03-05T09:14:27Z

Hey @jaraco! Did you come around to implementing the said interface? Maybe #4575 have some effect on this issue?

jaraco · 2018-03-05T13:27:35Z

@pradyunsg I haven't. It's still ticketed as pypa/setuptools#817. I've had little time to commit to advancing setuptools. If you could drum up some enthusiasm on that ticket to the point that someone suggests a patch, I'd be all for that.

pfmoore · 2018-03-05T14:04:50Z

Is there any reason this couldn't simply be

def get_provider(moduleOrReq, include_hidden=True):
    """Return an IResourceProvider for the named module or requirement"""
    if isinstance(moduleOrReq, Requirement):
        return working_set.find(moduleOrReq) or (include_hidden and require(str(moduleOrReq))[0])

so get_provider defaults to including hidden, but can be asked to ignore them? The call chain would of course need to be followed back up through to get_distribution, and pip would need to call get_distribution(PKGNAME, include_hidden=False).

I don't have time to make this into a full-fledged patch at the moment, but if anyone wants to they are welcome to the idea 😄

DachuanZhao · 2019-10-16T11:19:48Z

For people struggling with hack to get it right, I had similar issue:
nishnik@nishnik-Lenovo-Y50-70:~$ pip uninstall selenium
Cannot uninstall requirement selenium, not installed
nishnik@nishnik-Lenovo-Y50-70:~$ sudo pip3 install selenium --user
Requirement already satisfied (use --upgrade to upgrade): selenium in /usr/local/lib/python3.5/dist-packages
I deleted the directory /usr/local/lib/python3.5/dist-packages/selenium and then pip installed it again, it worked right for me.

It works ! Thanks !

pradyunsg · 2020-02-05T19:04:10Z

Blocked on pypa/setuptools#817.

AndydeCleyre · 2020-03-09T19:54:03Z

Is this the same issue, or something else?

$ mkdir piptrub
$ cd piptrub
$ git clone [email protected]:jazzband/pip-tools
$ cd pip-tools
$ python3 -m venv ../venv
$ . ../venv/bin/activate
$ pip install -U pip

$ pip --version
pip 20.0.2 from /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages/pip (python 3.8)
$ python -m pip --version
pip 20.0.2 from /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages/pip (python 3.8)
$ python -V
Python 3.8.2
$ which python
/home/andy/Code/piptrubbl/venv/bin/python

$ pip install .
$ pip uninstall -y pip-tools
$ python -m pip install pip-tools
Requirement already satisfied: pip-tools in /home/andy/Code/piptrubbl/pip-tools (4.5.2.dev18+g1d0883f)
Requirement already satisfied: click>=7 in /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages (from pip-tools) (7.1.1)
Requirement already satisfied: six in /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages (from pip-tools) (1.14.0)
Requirement already satisfied: pip>=20.0 in /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages (from pip-tools) (20.0.2)
$ pip freeze
click==7.1.1
six==1.14.0
$ pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-4.5.1-py2.py3-none-any.whl (41 kB)
Requirement already satisfied: click>=7 in /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages (from pip-tools) (7.1.1)
Requirement already satisfied: six in /home/andy/Code/piptrubbl/venv/lib/python3.8/site-packages (from pip-tools) (1.14.0)
Installing collected packages: pip-tools
Successfully installed pip-tools-4.5.1

jaraco · 2020-03-09T22:30:18Z

Is this the same issue, or something else?

After reading through it, I think the issue you're illustrating is that after pip install . on a checked out source and then uninstalling it, it still shows as installed. Interestingly, I don't get the same result as you did. I get instead what I think you expected:

draft $ cat pip499.sh
mkdir piptrub
cd piptrub
git clone [email protected]:jazzband/pip-tools
cd pip-tools
python3 -m venv ../venv
. ../venv/bin/activate
pip install -U pip
pip --version
python -m pip --version
python -V
which python
pip install .
pip uninstall -y pip-tools
python -m pip install pip-tools
pip freeze
pip install pip-tools
draft $ bash -x pip499.sh
+ mkdir piptrub
+ cd piptrub
+ git clone [email protected]:jazzband/pip-tools
Cloning into 'pip-tools'...
remote: Enumerating objects: 89, done.
remote: Counting objects: 100% (89/89), done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 6080 (delta 42), reused 39 (delta 16), pack-reused 5991
Receiving objects: 100% (6080/6080), 2.53 MiB | 5.73 MiB/s, done.
Resolving deltas: 100% (3602/3602), done.
+ cd pip-tools
+ python3 -m venv ../venv
+ . ../venv/bin/activate
...
+ pip install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2
+ pip --version
pip 20.0.2 from /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages/pip (python 3.8)
+ python -m pip --version
pip 20.0.2 from /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages/pip (python 3.8)
+ python -V
+ which python
Python 3.8.1
/Users/jaraco/draft/piptrub/venv/bin/python
+ pip install .
Processing /Users/jaraco/draft/piptrub/pip-tools
Collecting click>=7
  Using cached click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: pip>=20.0 in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (from pip-tools==4.5.2.dev18+g1d0883f) (20.0.2)
Installing collected packages: click, six, pip-tools
    Running setup.py install for pip-tools ... done
Successfully installed click-7.1.1 pip-tools-4.5.2.dev18+g1d0883f six-1.14.0
+ pip uninstall -y pip-tools
Found existing installation: pip-tools 4.5.2.dev18+g1d0883f
Uninstalling pip-tools-4.5.2.dev18+g1d0883f:
  Successfully uninstalled pip-tools-4.5.2.dev18+g1d0883f
+ python -m pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-4.5.1-py2.py3-none-any.whl (41 kB)
Requirement already satisfied: six in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (from pip-tools) (1.14.0)
Requirement already satisfied: click>=7 in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (from pip-tools) (7.1.1)
Installing collected packages: pip-tools
Successfully installed pip-tools-4.5.1
+ pip freeze
click==7.1.1
pip-tools==4.5.1
six==1.14.0
+ pip install pip-tools
Requirement already satisfied: pip-tools in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (4.5.1)
Requirement already satisfied: six in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (from pip-tools) (1.14.0)
Requirement already satisfied: click>=7 in /Users/jaraco/draft/piptrub/venv/lib/python3.8/site-packages (from pip-tools) (7.1.1)

So there's definitely something different between our two environments.

It's still ticketed as pypa/setuptools#817

I've just closed that ticket. I don't believe investment in pkg_resources is the right place for this issue. Instead, the latest importlib_metadata presents any number of Distribution objects. Probably if importlib_metadata.distributions(name='dist-name') returns more than one distribution for 'dist-name', pip should ascertain the best one to uninstall, ask the user which one to uninstall, or just raise an error indicating that the user's request is ambiguous.

AndydeCleyre · 2020-03-10T05:11:49Z

@jaraco

Well I'm having trouble reproducing now using that same procedure, but I think the trouble was actually a duplicate of #6558.

EDIT: Ah, I wasn't reproducing because I need to run pip install -e . somewhere along the way in order to generate that local egg-info folder. So the following will always reproduce, but yeah it's #6558.

$ mkdir piptrub
$ cd piptrub
$ git clone [email protected]:jazzband/pip-tools
$ cd pip-tools
$ python3 -m venv ../venv
$ . ../venv/bin/activate
$ pip install -U pip
$ pip install -e .
$ pip uninstall -y pip-tools
$ python -m pip install pip-tools

It is surprising that python -m pip install will erroneously consider the req already satisfied, but pip install won't.

sbidoul · 2020-04-05T10:06:44Z

This should be resolved by #7955.

jaraco mentioned this issue Oct 17, 2016

pkg_resources should present an interface for reporting installed packages pypa/setuptools#817

Closed

pradyunsg added type: enhancement Improvements to functionality S: awaiting response Waiting for a response/more information labels Mar 5, 2018

pradyunsg removed the S: awaiting response Waiting for a response/more information label May 30, 2018

pradyunsg added the state: blocked Can not be done until something else is done label Feb 5, 2020

deveshks mentioned this issue Apr 5, 2020

Don't list packages in current directory #7955

Merged

pradyunsg closed this as completed in #7955 Apr 5, 2020

deveshks mentioned this issue Apr 5, 2020

Add unit tests for pip commands not using cwd #7987

Merged

deveshks mentioned this issue Apr 18, 2020

Fix incorrect comment in tests in test_install #8012

Merged

lock bot added the auto-locked Outdated issues that have been locked by automation label May 5, 2020

lock bot locked as resolved and limited conversation to collaborators May 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Requirement already satisfied by egg but package not installed #3710

Requirement already satisfied by egg but package not installed #3710

jaraco commented May 21, 2016

dstufft commented Sep 7, 2016

jaraco commented Sep 7, 2016

dstufft commented Sep 7, 2016

jaraco commented Sep 7, 2016

jaraco commented Sep 16, 2016

dstufft commented Sep 16, 2016

xavfernandez commented Oct 14, 2016

nishnik commented Nov 13, 2016

pradyunsg commented Mar 5, 2018

jaraco commented Mar 5, 2018

pfmoore commented Mar 5, 2018 •

edited

Loading

DachuanZhao commented Oct 16, 2019

pradyunsg commented Feb 5, 2020

AndydeCleyre commented Mar 9, 2020

jaraco commented Mar 9, 2020

AndydeCleyre commented Mar 10, 2020 •

edited

Loading

sbidoul commented Apr 5, 2020

Requirement already satisfied by egg but package not installed #3710

Requirement already satisfied by egg but package not installed #3710

Comments

jaraco commented May 21, 2016

Description:

What I've run:

dstufft commented Sep 7, 2016

jaraco commented Sep 7, 2016

dstufft commented Sep 7, 2016

jaraco commented Sep 7, 2016

jaraco commented Sep 16, 2016

dstufft commented Sep 16, 2016

xavfernandez commented Oct 14, 2016

nishnik commented Nov 13, 2016

pradyunsg commented Mar 5, 2018

jaraco commented Mar 5, 2018

pfmoore commented Mar 5, 2018 • edited Loading

DachuanZhao commented Oct 16, 2019

pradyunsg commented Feb 5, 2020

AndydeCleyre commented Mar 9, 2020

jaraco commented Mar 9, 2020

AndydeCleyre commented Mar 10, 2020 • edited Loading

sbidoul commented Apr 5, 2020

pfmoore commented Mar 5, 2018 •

edited

Loading

AndydeCleyre commented Mar 10, 2020 •

edited

Loading