Connection timed out failure in pip when using proxy

Wednesday, October 15th 2014, 8:49 pm

Today a friend and I were working on installing a Python application on a server that sat behind an HTTP proxy. The first step was figuring out how to tell pip to use an http proxy. This step was simple, we just had to add the proxy parameter on the commandline, like so:

$ pip install -r requirements.txt --proxy=<proxy address>

The installation seemed to be proceeding correctly, with pip telling us files were getting downloaded succesfully from pypi.python.org. Unfortunately about halfway through the installation pip aborted with the following message:

File "/usr/local/lib/python2.7/site-packages/`pip`-1.5.5-py2.7.egg/pip/_vendor/requests/adapters.py", line 387, in send
    raise Timeout(e)
Timeout: (<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x1f70650>, 'Connection to pypi.python.org timed out. (connect timeout=15)')

We reasoned there had to be something wrong with either our environment or with the proxy and thus we proceeded to waste the following hour debugging and checking different parts of the system for problems.

Imagine our surprise when we decided to search online and ran into this bug: https://github.com/pypa/pip/issues/1805. Apparently this is a know issue in urllib. pip, through requests seems to use a pool of HTTP connections to download the different packages from the internet. When there is a large number of packages, new connections in the pool are created, however they do not seem to inherit the proxy settings. The result is that the connection times out. This was discovered by github user alexandrem. In his comment inside the bug he proposes a small fix:

Fix for timeout problem
Open file: ../lib/python2.6/site-packages/pip/_vendor/requests/adapters.py
Change line 209 to the following:
if True or not proxy in self.proxy_manager:

This effectively forces pip to instantiate a new ProxyManager object every time, but it allows the installation to complete. I thought it was a fair tradeoff, with no obvious side-effects.

There is a fix already in the requests package, but it doesn't seem to have reached the latest pip. Until it does, you can use the previous fix, to allow your installations to complete succesfully.

comments powered byDisqus