Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with 301-redirect #1360

Closed
Cosmologist opened this issue May 16, 2013 · 4 comments · Fixed by #1361 · May be fixed by #6923
Closed

Problem with 301-redirect #1360

Cosmologist opened this issue May 16, 2013 · 4 comments · Fixed by #1361 · May be fixed by #6923

Comments

@Cosmologist
Copy link

I have a problem with downloading a file, when the server returns the 301 code:

r = requests.get('http://servag.rupsy.ru/track.php?url=2013-04-11/Haatzumaniak_-_Dark_Sadhu_[2013]/03.%20Haatzumaniak_-_777_Dark%20Sadhu%202013.mp3&arj=Haatzumaniak_-_Dark_Sadhu_[2013].ZIP')
print(r)

If you do redirect manually - it works fine:

r = requests.get('http://servag.rupsy.ru/track.php?url=2013-04-11/Haatzumaniak_-_Dark_Sadhu_[2013]/03.%20Haatzumaniak_-_777_Dark%20Sadhu%202013.mp3&arj=Haatzumaniak_-_Dark_Sadhu_[2013].ZIP', allow_redirects=False)
r = requests.get(r.headers['Location'])
print(r)

The problem appears to Python 3.3 and python-requests-1.2.0/python-requests-dev

Traceback (most recent call last):
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 428, in urlopen
    body=body, headers=headers)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 290, in _make_request
    httplib_response = conn.getresponse()
  File "c:\Python33\lib\http\client.py", line 1143, in getresponse
    response.begin()
  File "c:\Python33\lib\http\client.py", line 354, in begin
    version, status, reason = self._read_status()
  File "c:\Python33\lib\http\client.py", line 324, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\adapters.py", line 283, in send
    timeout=timeout
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 474, in urlopen
    raise MaxRetryError(self, url, e)
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='servag.rupsy.ru', port=80): Max retries exceeded with url: /media/2/Haatzumaniak_-_Dark_Sadhu_[2013]/03. Haatzumaniak_-_777_Dark Sadhu 2013.mp3 (Caused by <class 'http.client.BadStatusLine'>: '')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Dropbox/projects/InfoStreams/test.py", line 25, in <module>
    allow_redirects=True)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\sessions.py", line 345, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\sessions.py", line 464, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\sessions.py", line 464, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\sessions.py", line 154, in resolve_redirects
    allow_redirects=False,
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\sessions.py", line 448, in send
    r = adapter.send(request, **kwargs)
  File "C:\Dropbox\projects\Helpers\ph\.envwin32\lib\site-packages\requests\adapters.py", line 318, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='servag.rupsy.ru', port=80): Max retries exceeded with url: /media/2/Haatzumaniak_-_Dark_Sadhu_[2013]/03. Haatzumaniak_-_777_Dark Sadhu 2013.mp3 (Caused by <class 'http.client.BadStatusLine'>: '')
@Lukasa
Copy link
Member

Lukasa commented May 16, 2013

Uh...can I ask you to uninstall and reinstall Requests? The TypeError that got thrown should have been impossible to hit, because it's inside a Try/Catch block that catches TypeErrors.

@Cosmologist
Copy link
Author

Ignore the first trace (I'll delete it), the key of the problem is:

File "c:\Python33\lib\http\client.py", line 324, in _read_status
    raise BadStatusLine(line)

@Lukasa
Copy link
Member

Lukasa commented May 16, 2013

Ah, ok, got it.

The issue is that the remote server is not following RFC 2616. The Location header should be percent-encoded, and hasn't been. This doesn't matter if you pass the URL through Requests from the top, because it gets percent-encoded by prepare_url.

EDIT: To be clear, we should handle this: I'll have a fix ready shortly.

@Lukasa Lukasa mentioned this issue May 16, 2013
Merged
@Cosmologist
Copy link
Author

Lucasa, thank you very much - now works fine

@Lukasa Lukasa mentioned this issue Nov 25, 2014
Closed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 9, 2021
moben added a commit to moben/requests that referenced this issue Mar 28, 2025
@moben
Don't normalise or double-escape urls
a8086f8 
`requote_uri` was added in 2013 in psf#1361 to resolve psf#1360.

But in 2019 this was also solved in `urllib3` in
urllib3/urllib3#1647.

Not only does the `urllib3` implementation more closely match what
all other clients are doing.  It also means that we are double-encoding
invalid uris.  If the redirect is invalid we're in unchartered territory
but the path that `requests` uses has no chance to decode to the same
that any other client uses, even to the most lenient server.

Simply drop `requote_uri`, as `urllib3`'s `parse_url` will always handle
invalid urls.
moben added a commit to moben/requests that referenced this issue Mar 28, 2025
@moben
Don't normalise or double-escape urls
7ec84f3 
`requote_uri` was added in 2013 in psf#1361 to resolve psf#1360.

But in 2019 this was also solved in `urllib3` in
urllib3/urllib3#1647.

Not only does the `urllib3` implementation more closely match what
all other clients are doing.  It also means that we are double-encoding
invalid uris.  If the redirect is invalid we're in unchartered territory
but the path that `requests` uses has no chance to decode to the same
that any other client uses, even to the most lenient server.

Simply drop `requote_uri`, as `urllib3`'s `parse_url` will always handle
invalid urls.
moben added a commit to moben/requests that referenced this issue Mar 29, 2025
@moben
Don't normalise or double-escape urls
3a47592 
`requote_uri` was added in 2013 in psf#1361 to resolve psf#1360.

But in 2019 this was also solved in `urllib3` in
urllib3/urllib3#1647.

Not only does the `urllib3` implementation more closely match what
all other clients are doing.  It also means that we are double-encoding
invalid uris.  If the redirect is invalid we're in unchartered territory
but the path that `requests` uses has no chance to decode to the same
that any other client uses, even to the most lenient server.

Simply drop `requote_uri`, as `urllib3`'s `parse_url` will always handle
invalid urls.
moben added a commit to moben/requests that referenced this issue Mar 29, 2025
@moben
Don't normalise or double-escape urls
0aceb9f 
`requote_uri` was added in 2013 in psf#1361 to resolve psf#1360.

But in 2019 this was also solved in `urllib3` in
urllib3/urllib3#1647.

Not only does the `urllib3` implementation more closely match what
all other clients are doing.  It also means that we are double-encoding
invalid uris.  If the redirect is invalid we're in unchartered territory
but the path that `requests` uses has no chance to decode to the same
that any other client uses, even to the most lenient server.

Simply drop `requote_uri`, as `urllib3`'s `parse_url` will always handle
invalid urls.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@Cosmologist @Lukasa