Skip to content
This repository was archived by the owner on Dec 17, 2019. It is now read-only.

Rudy/master updated #2

Open
wants to merge 37 commits into
base: datadog
Choose a base branch
from
Open

Rudy/master updated #2

wants to merge 37 commits into from

Conversation

bunelr
Copy link

@bunelr bunelr commented Aug 29, 2014

Merged the tip of the original repo into our branch, and resolved the merge conflicts so that our code doesn't have to be changed, but that makes it different from the main repo because they put some backwards incompatible changes.
It made the switch from urllib2 to provide the connection to requests (so urllib3 inside).

On top of that, I changed the connection to pingdom api to use a request.Session instead of simple requests request which allows to cache the connection and give a small speed boost.

From a profile run from our crawler, that spends most of the time doing the http calls:

Before:

build/bdist.linux-x86_64/egg/pingdom/connection.py:87(fetch) ->    516    0.006  495.193  /usr/lib/python2.7/urllib2.py:122(urlopen

After:

build/bdist.linux-x86_64/egg/pingdom/connection.py:71(fetch) ->    520    0.010  370.924  /home/vagrant/dogweb/python/local/lib/python2.7/site-packages/requests-2.2.0-py2.7.egg/requests/sessions.py:387(get)

516/520 is the number of times the http cales are made. 495/370 is the cumulative time spent doing them.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants