pip install --user -U <package> fails to upgrade packages #5174

mgedmin · 2018-04-05T16:19:10Z

Pip version: 10.0.0b2
Python version: 2.7.14
Operating system: Ubuntu 17.10

Description:

I ran pip list --user --outdated, saw that I had freezegun 0.3.7 in my ~/.local/lib/python2.7/site-packages, and tried to upgrade it with pip install --user -U freezegun. This did not work:

Requirement already satisfied: freezegun in /home/mg/.local/lib/python2.7/site-packages (0.3.7)

Then I downgraded pip to 9.0.3 with pip install "pip<10" and tried the pip install --user -U freezegun again, and it successfully upgraded the freezegun ... I think?

Cache entry deserialization failed, entry ignored
Collecting freezegun
  Using cached freezegun-0.3.10-py2.py3-none-any.whl
Cache entry deserialization failed, entry ignored
Requirement already up-to-date: six in /home/mg/.venv/lib/python2.7/site-packages (from freezegun)
Collecting python-dateutil!=2.0,>=1.0 (from freezegun)
  Using cached python_dateutil-2.7.2-py2.py3-none-any.whl
Installing collected packages: python-dateutil, freezegun
  Found existing installation: freezegun 0.3.7
    Not uninstalling freezegun at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
Successfully installed freezegun-0.3.10 python-dateutil-2.7.2
You are using pip version 9.0.3, however version 10.0.0b2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(Despite the scary "not uninstalling ..., outside environment" warning, pip list --user confirms that I've freezegun 0.3.10 in my ~/.local/lib/python2.7/site-packages now.)

The text was updated successfully, but these errors were encountered:

mgedmin · 2018-04-05T16:29:35Z

Hm, I cannot reproduce this with a different package (pov-fabric-helpers):

$ pip install --user -U pov-fabric-helpers
Collecting pov-fabric-helpers
  Downloading pov_fabric_helpers-0.3-py2-none-any.whl
Requirement not upgraded as not directly required: Fabric in /home/mg/.venv/lib/python2.7/site-packages (from pov-fabric-helpers) (1.14.0)
Requirement not upgraded as not directly required: paramiko<3.0,>=1.10 in /home/mg/.venv/lib/python2.7/site-packages (from Fabric->pov-fabric-helpers) (2.4.1)
Requirement not upgraded as not directly required: cryptography>=1.5 in /home/mg/.venv/lib/python2.7/site-packages (from paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (2.2.2)
Requirement not upgraded as not directly required: pynacl>=1.0.1 in /home/mg/.venv/lib/python2.7/site-packages (from paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (1.2.1)
Requirement not upgraded as not directly required: pyasn1>=0.1.7 in /home/mg/.venv/lib/python2.7/site-packages (from paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (0.4.2)
Requirement not upgraded as not directly required: bcrypt>=3.1.3 in /home/mg/.venv/lib/python2.7/site-packages (from paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (3.1.4)
Requirement not upgraded as not directly required: cffi>=1.7; platform_python_implementation != "PyPy" in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (1.11.5)
Requirement not upgraded as not directly required: enum34; python_version < "3" in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (1.1.6)
Requirement not upgraded as not directly required: idna>=2.1 in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (2.6)
Requirement not upgraded as not directly required: asn1crypto>=0.21.0 in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (0.24.0)
Requirement not upgraded as not directly required: six>=1.4.1 in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (1.11.0)
Requirement not upgraded as not directly required: ipaddress; python_version < "3" in /home/mg/.venv/lib/python2.7/site-packages (from cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (1.0.19)
Requirement not upgraded as not directly required: pycparser in /home/mg/.venv/lib/python2.7/site-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.5->paramiko<3.0,>=1.10->Fabric->pov-fabric-helpers) (2.18)
acme 0.23.0 requires mock, which is not installed.
acme 0.23.0 requires pyrfc3339, which is not installed.
acme 0.23.0 requires pytz, which is not installed.
certbot-apache 0.23.0 requires mock, which is not installed.
pytest 3.5.0 requires funcsigs; python_version < "3.0", which is not installed.
readme-renderer 18.1 requires future, which is not installed.
ansible-cmdb 1.26.1 requires mako, which is not installed.
requests 2.18.4 requires chardet<3.1.0,>=3.0.2, which is not installed.
imgdiff 1.7.0 requires Pillow, which is not installed.
certbot 0.23.0 requires configobj, which is not installed.
certbot 0.23.0 requires mock, which is not installed.
certbot 0.23.0 requires parsedatetime>=1.3, which is not installed.
certbot 0.23.0 requires pyrfc3339, which is not installed.
certbot 0.23.0 requires pytz, which is not installed.
Installing collected packages: pov-fabric-helpers
  Found existing installation: pov-fabric-helpers 0.2
    Not uninstalling pov-fabric-helpers at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
    Can't uninstall 'pov-fabric-helpers'. No files were found to uninstall.
Successfully installed pov-fabric-helpers-0.3

Now I'm using 10.0.0b2 installed from PyPI. When I filed this bug I was using 10.0.0b2 from a git snapshot mentioned in #5160 (comment).

pfmoore · 2018-04-05T17:14:22Z

@pradyunsg ping - is this likely to be related to your fix #5162?

pradyunsg · 2018-04-05T17:39:50Z

I don't think so. That PR doesn't affect what gets installed.

mgedmin · 2018-04-05T18:23:32Z

I've downgraded freezegun back to 0.3.7 (using pip 0.9.3), upgraded pip to 10.0.0b2 with pip install -U --pre pip, and reproduced the original issue:

~ $ pip install --user -U freezegun
Requirement already up-to-date: freezegun in ./.local/lib/python2.7/site-packages (0.3.10)
Requirement not upgraded as not directly required: six in ./.venv/lib/python2.7/site-packages (from freezegun) (1.11.0)
Requirement not upgraded as not directly required: python-dateutil!=2.0,>=1.0 in ./.local/lib/python2.7/site-packages (from freezegun) (2.7.2)
acme 0.23.0 requires mock, which is not installed.
acme 0.23.0 requires pyrfc3339, which is not installed.
acme 0.23.0 requires pytz, which is not installed.
certbot-apache 0.23.0 requires mock, which is not installed.
pytest 3.5.0 requires funcsigs; python_version < "3.0", which is not installed.
readme-renderer 18.1 requires future, which is not installed.
ansible-cmdb 1.26.1 requires mako, which is not installed.
requests 2.18.4 requires chardet<3.1.0,>=3.0.2, which is not installed.
imgdiff 1.7.0 requires Pillow, which is not installed.
certbot 0.23.0 requires configobj, which is not installed.
certbot 0.23.0 requires mock, which is not installed.
certbot 0.23.0 requires parsedatetime>=1.3, which is not installed.
certbot 0.23.0 requires pyrfc3339, which is not installed.
certbot 0.23.0 requires pytz, which is not installed.

(Ignore the "requires X which is not installed" errors, those are #5160.)

So there must be something different between freezegun and pip-fabric-helpers? Perhaps there's another package that depends on freezegun, but no packages that depend on pov-fabric-helpers?

mgedmin · 2018-04-05T18:30:18Z

I do not know why pip 10 thinks freezegun is at version 0.3.10. Is there some cache that got stale when pip 9 downgraded freezegun? If I check /home/mg/.local/lib/python2.7/site-packages/freezegun/__init__.py, I see

__version__ = '0.3.7'

pradyunsg · 2018-04-05T18:36:40Z

What does pip freeze show?

mgedmin · 2018-04-05T18:37:16Z

$ pip freeze|grep freezegun
freezegun==0.3.10

pradyunsg · 2018-04-05T18:37:57Z

What about pip 9?

mgedmin · 2018-04-05T18:38:28Z

oh hey, pip 9 also thinks the version is 0.3.10?!?

$ pip freeze|grep freezegun
freezegun==0.3.10

mgedmin · 2018-04-05T18:39:04Z

I still have the original downgrade command in my scrollback. It was

$ pip install --user freezegun==0.3.7
Collecting freezegun==0.3.7
Requirement already satisfied: six in ./.venv/lib/python2.7/site-packages (from freezegun==0.3.7)
Requirement already satisfied: python-dateutil>=2.0 in ./.local/lib/python2.7/site-packages (from freezegun==0.3.7)
Installing collected packages: freezegun
  Found existing installation: freezegun 0.3.10
    Not uninstalling freezegun at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
Successfully installed freezegun-0.3.10
You are using pip version 9.0.3, however version 10.0.0b2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

mgedmin · 2018-04-05T18:40:39Z

Hmm

$ find ~/.venv/lib/ ~/.local/lib/python* -name 'freezegun*'
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.7.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.10.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun

mgedmin · 2018-04-05T18:42:40Z

Let's see what happens if I clean this up

mg@platonas: ~ $ rm -r /home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.10.dist-info

mg@platonas: ~ $ pip freeze|grep freezegun
freezegun==0.3.7
You are using pip version 9.0.3, however version 10.0.0b2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

mg@platonas: ~ $ pip install -U --pre pip
Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached pip-10.0.0b2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.0b2

And now trying to reproduce the original issue

mg@platonas: ~ $ pip install --user -U freezegun
Collecting freezegun
  Using cached freezegun-0.3.10-py2.py3-none-any.whl
Requirement not upgraded as not directly required: six in ./.venv/lib/python2.7/site-packages (from freezegun) (1.11.0)
Requirement not upgraded as not directly required: python-dateutil!=2.0,>=1.0 in ./.local/lib/python2.7/site-packages (from freezegun) (2.7.2)
freezegun 0.3.10 requires python-dateutil!=2.0,>=1.0, which is not installed.
acme 0.23.0 requires mock, which is not installed.
acme 0.23.0 requires pyrfc3339, which is not installed.
acme 0.23.0 requires pytz, which is not installed.
certbot-apache 0.23.0 requires mock, which is not installed.
pytest 3.5.0 requires funcsigs; python_version < "3.0", which is not installed.
readme-renderer 18.1 requires future, which is not installed.
ansible-cmdb 1.26.1 requires mako, which is not installed.
requests 2.18.4 requires chardet<3.1.0,>=3.0.2, which is not installed.
imgdiff 1.7.0 requires Pillow, which is not installed.
certbot 0.23.0 requires configobj, which is not installed.
certbot 0.23.0 requires mock, which is not installed.
certbot 0.23.0 requires parsedatetime>=1.3, which is not installed.
certbot 0.23.0 requires pyrfc3339, which is not installed.
certbot 0.23.0 requires pytz, which is not installed.
Installing collected packages: freezegun
  Found existing installation: freezegun 0.3.7
    Not uninstalling freezegun at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
    Can't uninstall 'freezegun'. No files were found to uninstall.
Successfully installed freezegun-0.3.10

Cannot reproduce it!

However

mg@platonas: ~ $ find ~/.venv/lib/ ~/.local/lib/python* -name 'freezegun*'
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.7.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.10.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun

seems wrong.

pfmoore · 2018-04-05T18:46:34Z

Yeah, that looks wrong. Probably caused by doing a --user install with a virtualenv activated? (That's likely what caused the "not uninstalling ..., outside environment" warning - it installed the new version, but didn't uninstall the old version because it's not in the active virtualenv. (note the line "Requirement already up-to-date: six in /home/mg/.venv/lib/python2.7/site-packages (from freezegun)" in the original traceback).

Basically #5176 that you raised earlier, rearing its ugly head when you're doing an upgrade (which is uninstall + install, effectively).

mgedmin · 2018-04-05T18:57:20Z

I cannot reproduce this even after installing the snapshot again:

mg@platonas: ~ $ rm -r /home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.7.dist-info

mg@platonas: ~ $ pip install --user freezegun==0.3.7Collecting freezegun==0.3.7
Requirement already satisfied: six in ./.venv/lib/python2.7/site-packages (from freezegun==0.3.7) (1.11.0)
Requirement already satisfied: python-dateutil>=2.0 in ./.local/lib/python2.7/site-packages (from freezegun==0.3.7) (2.7.2)
freezegun 0.3.7 requires python-dateutil>=2.0, which is not installed.
acme 0.23.0 requires mock, which is not installed.
acme 0.23.0 requires pyrfc3339, which is not installed.
acme 0.23.0 requires pytz, which is not installed.
certbot-apache 0.23.0 requires mock, which is not installed.
pytest 3.5.0 requires funcsigs; python_version < "3.0", which is not installed.
readme-renderer 18.1 requires future, which is not installed.
ansible-cmdb 1.26.1 requires mako, which is not installed.
requests 2.18.4 requires chardet<3.1.0,>=3.0.2, which is not installed.
imgdiff 1.7.0 requires Pillow, which is not installed.
certbot 0.23.0 requires configobj, which is not installed.
certbot 0.23.0 requires mock, which is not installed.
certbot 0.23.0 requires parsedatetime>=1.3, which is not installed.
certbot 0.23.0 requires pyrfc3339, which is not installed.
certbot 0.23.0 requires pytz, which is not installed.
Installing collected packages: freezegun
  Found existing installation: freezegun 0.3.10
    Not uninstalling freezegun at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
    Can't uninstall 'freezegun'. No files were found to uninstall.
Successfully installed freezegun-0.3.10

mg@platonas: ~ $ find ~/.venv/lib/ ~/.local/lib/python* -name 'freezegun*'
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.7.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.10.dist-info
/home/mg/.local/lib/python2.7/site-packages/freezegun

mg@platonas: ~ $ less /home/mg/.local/lib/python2.7/site-packages/freezegun/__init__.py

mg@platonas: ~ $ rm -r /home/mg/.local/lib/python2.7/site-packages/freezegun-0.3.10.dist-info

mg@platonas: ~ $ grep __version__ /home/mg/.local/lib/python2.7/site-packages/freezegun/__init__.py
__version__ = '0.3.7'

mg@platonas: ~ $ pip install -U https://github.com/pypa/pip/archive/release/10.0.0.zip
Collecting https://github.com/pypa/pip/archive/release/10.0.0.zip
  Downloading https://github.com/pypa/pip/archive/release/10.0.0.zip
     - 6.4MB 1.4MB/s
  Installing build dependencies ... done
Building wheels for collected packages: pip
  Running setup.py bdist_wheel for pip ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-AJ1LKH/wheels/96/44/b1/106b04e7cb81d2106b30cf539ded898ef9dab715860495fea9
Successfully built pip
acme 0.23.0 requires mock, which is not installed.
acme 0.23.0 requires pyrfc3339, which is not installed.
acme 0.23.0 requires pytz, which is not installed.
certbot-apache 0.23.0 requires mock, which is not installed.
pytest 3.5.0 requires funcsigs; python_version < "3.0", which is not installed.
readme-renderer 18.1 requires future, which is not installed.
ansible-cmdb 1.26.1 requires mako, which is not installed.
requests 2.18.4 requires chardet<3.1.0,>=3.0.2, which is not installed.
imgdiff 1.7.0 requires Pillow, which is not installed.
certbot 0.23.0 requires configobj, which is not installed.
certbot 0.23.0 requires mock, which is not installed.
certbot 0.23.0 requires parsedatetime>=1.3, which is not installed.
certbot 0.23.0 requires pyrfc3339, which is not installed.
certbot 0.23.0 requires pytz, which is not installed.
Installing collected packages: pip
  Found existing installation: pip 10.0.0b2
    Uninstalling pip-10.0.0b2:
      Successfully uninstalled pip-10.0.0b2
Successfully installed pip-10.0.0b2

[15s]
mg@platonas: ~ $ pip install --user -U freezegun
Collecting freezegun
  Using cached freezegun-0.3.10-py2.py3-none-any.whl
Requirement not upgraded as not directly required: six in ./.venv/lib/python2.7/site-packages (from freezegun) (1.11.0)
Requirement not upgraded as not directly required: python-dateutil!=2.0,>=1.0 in ./.local/lib/python2.7/site-packages (from freezegun) (2.7.2)
Installing collected packages: freezegun
  Found existing installation: freezegun 0.3.7
    Not uninstalling freezegun at /home/mg/.local/lib/python2.7/site-packages, outside environment /home/mg/.venv
    Can't uninstall 'freezegun'. No files were found to uninstall.
Successfully installed freezegun-0.3.10

I cannot explain my original experience. :/

pradyunsg · 2018-04-06T06:49:01Z

Do you have an activated venv?

mgedmin · 2018-04-06T07:12:35Z

It's not activated per se (it's not in $PYTHONPATH), but its bin/ directory is early in my $PATH.

pfmoore · 2018-04-06T07:54:44Z

@mgedmin Being on PATH is all that activation involves (in practical terms) so that's going to be the issue here (the pip command you are running is probably coming from that virtualenv - pip --version should confirm this).

mgedmin · 2018-04-06T08:10:51Z

pip is absolutely from that virtualenv (via a wrapper script, which should not matter).

(The purpose of that virtualenv is so I can use pip to put tools on my $PATH, without having to use sudo.
I think back when I came up with this scheme pip install --user didn't exist.)

pfmoore · 2018-04-06T08:16:35Z

OK. So that's your issue - "not uninstalling ..., outside environment" is because you're using the pip from that environment and it is not able to uninstall code that wasn't installed into that environment's site-packages. Hence the brokenness on upgrades.

mgedmin · 2018-04-06T09:12:26Z

Yeah, #5176 basically.

(It's a bit strange that pip from an activated virtualenv can happily pip install --user stuff into ~/.local/lib/pythonX.Y, but it can't uninstall.)

But the original point of this bug was not that pip botched the upgrade by not removing the stale dist-info. It was that pip refused to even consider the upgrade. If only I could repro... hey.

Now that I remember, what I did was write a shell script pip-outdated, ran it with --user, saw a bunch of outdated packages, decided to upgrade three of them in one go with

pip install --user -U freezegun pov-fabric-helpers tornado

and then got that original error.

Could it be that freezegun is a dependency of, say, tornado? Could it be that pip 10 refuses to upgrade packages that are dependencies of other packages, even if you explicitly ask for both packages to be upgraded?

Further checking (grep freezegun /home/mg/.local/lib/python2.7/site-packages/*.dist-info/METADATA) says no, nothing in my ~/.local/lib/ requires freezegun. (Well, pip does, as part of the testing extra. And gtimelog, as part of the test extra. And this raises questions: why do I have them in ~/.local/lib/? Both pip and gtimelog are supposed to be installed in ~/.venv/lib!)

I think it may be best to close this bug report, since I can't reproduce the original issue, and the side issues encountered during the various experiments have their own issues filed.

lock · 2019-05-29T00:03:13Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

dstufft added this to the 10.0 milestone Apr 5, 2018

dstufft added the !release blocker Hold a release until this is resolved label Apr 5, 2018

mgedmin closed this as completed Apr 6, 2018

pradyunsg removed !release blocker Hold a release until this is resolved labels Feb 8, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019

lock bot locked as resolved and limited conversation to collaborators May 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip install --user -U <package> fails to upgrade packages #5174

pip install --user -U <package> fails to upgrade packages #5174

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

pfmoore commented Apr 5, 2018

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018 •

edited

Loading

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018 •

edited

Loading

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

pfmoore commented Apr 5, 2018 •

edited

Loading

mgedmin commented Apr 5, 2018

pradyunsg commented Apr 6, 2018

mgedmin commented Apr 6, 2018

pfmoore commented Apr 6, 2018

mgedmin commented Apr 6, 2018

pfmoore commented Apr 6, 2018

mgedmin commented Apr 6, 2018

lock bot commented May 29, 2019

pip install --user -U <package> fails to upgrade packages #5174

pip install --user -U <package> fails to upgrade packages #5174

Comments

mgedmin commented Apr 5, 2018

Description:

mgedmin commented Apr 5, 2018

pfmoore commented Apr 5, 2018

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018 • edited Loading

pradyunsg commented Apr 5, 2018

mgedmin commented Apr 5, 2018 • edited Loading

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

mgedmin commented Apr 5, 2018

pfmoore commented Apr 5, 2018 • edited Loading

mgedmin commented Apr 5, 2018

pradyunsg commented Apr 6, 2018

mgedmin commented Apr 6, 2018

pfmoore commented Apr 6, 2018

mgedmin commented Apr 6, 2018

pfmoore commented Apr 6, 2018

mgedmin commented Apr 6, 2018

lock bot commented May 29, 2019

mgedmin commented Apr 5, 2018 •

edited

Loading

mgedmin commented Apr 5, 2018 •

edited

Loading

pfmoore commented Apr 5, 2018 •

edited

Loading