Skip to content

Instantly share code, notes, and snippets.

@Niccolum

Niccolum/readme.md

Last active March 16, 2020 07:14
Show Gist options
  • Save Niccolum/82f1169554eb4b8a77ed54371467a29e to your computer and use it in GitHub Desktop.
Save Niccolum/82f1169554eb4b8a77ed54371467a29e to your computer and use it in GitHub Desktop.
Download ZIP
history of popular python http clients
Raw
readme.md

sync

stdlib

httplib

1.4 1996-10-25

author Guido Van Rossum

This module defines a class which implements the client side of the HTTP protocol. It is normally not used directly - the module urllib uses it to handle URLs that use HTTP. https://docs.python.org/release/1.4/lib/node114.html#SECTION001230000000000000000

What can this library, the main things:
  • create connections
  • https support (ssl.SSLContext)
  • Errors
  • Request with http methods
  • Open/close connection and keep the connection open

urllib

1.4 1996-10-25

author Guido Van Rossum

The urllib module provides a simple interface for network resource access.

What can this library, the main things:
  • get requests (urlopen, urlretrieve, URLopener), only GET or (with special cases) POST
  • https support (ssl.SSLContext)
  • works with link (quote, quote_plus, unquote, unquote_plus, urlencode and etc)

urllib2

2.1 2001-04-15

author Jeremy Hylton

The urllib2 module provides an updated API for using internet resources identified by URLs. It is designed to be extended by individual applications to support new protocols or add variations to existing protocols (such as handling HTTP basic authentication).

What can this library, the main things:
  • urlopen with custom cacert and timeout
  • Errors
  • creating Request object, only GET or POST
  • handlers (HTTPDefaultErrorHandler, HTTPRedirectHandler, HTTPCookieProcessor and etc)
  • OpenerDirector for manage handler (add_handler), requests (open) and errors (error)
  • supports Basic, Digest auth

refactoring httplib -> http.client

3.0 2008-12-03

author Guido Van Rossum

Nothing major changes

refactoring urllib

3.0 2008-12-03

author Guido Van Rossum

urllib is a package that collects several modules for working with URLs:

  • urllib.request for opening and reading URLs

    What can this library, the main things:
    • urlopen
    • Request object with method (with 3.3, finally added other http methods)
    • handlers (HTTPDefaultErrorHandler, HTTPRedirectHandler, HTTPCookieProcessor and etc)
    • OpenerDirector for manage handler (add_handler), requests (open) and errors (error)
    • supports Basic, Digest auth

  • urllib.error containing the exceptions raised by urllib.request

    What can this library, the main things:
    • moved errors

  • urllib.parse for parsing URLs

    What can this library, the main things:
    • works with url parsing and quoting

  • urllib.robotparser for parsing robots.txt files

    • adding parser robot.txt

non-stdlib

httplib2

0.1 Rev 117 2006-02-07

0.17.0 2020-01-24 (latest release) "Development Status :: 4 - Beta"

author Joe Gregorio

The httplib2 module is a comprehensive HTTP client library that handles caching, keep-alive, compression, redirects and many kinds of authentication.

Writes only with python stdlib

Support versions:

  • "Programming Language :: Python",
  • "Programming Language :: Python :: 2",
  • "Programming Language :: Python :: 2.7",
  • "Programming Language :: Python :: 3",
  • "Programming Language :: Python :: 3.4",
  • "Programming Language :: Python :: 3.5",
  • "Programming Language :: Python :: 3.6",
  • "Programming Language :: Python :: 3.7",
What can this library, the main things:
  • Supports HTTPS
  • Supports HTTP 1.1 Keep-Alive
  • Supports several auth types: Digest, Basic, WSSE
  • Supports all methods
  • Supports a little of things, which not in stdlib, for example RedirectLimit, Compression, works with cache

urllib3

0.1 2008-11-16 (First release),

1.0 2011-10-08 (major version, production-ready)

1.25.8 2020-01-20 (latest release)

author Andrey Petrov

urllib3 is a powerful, sanity-friendly HTTP client for Python.

Writes only with python stdlib

Support versions:

  • "Programming Language :: Python",
  • "Programming Language :: Python :: 2",
  • "Programming Language :: Python :: 2.7",
  • "Programming Language :: Python :: 3",
  • "Programming Language :: Python :: 3.5",
  • "Programming Language :: Python :: 3.6",
  • "Programming Language :: Python :: 3.7",
  • "Programming Language :: Python :: 3.8",
  • "Programming Language :: Python :: 3.9",
  • "Programming Language :: Python :: Implementation :: CPython",
  • "Programming Language :: Python :: Implementation :: PyPy",
extras_require={
    "brotli": ["brotlipy"], #  Google zip algorithm
    "secure": [
        "pyOpenSSL",
        "cryptography",
        "idna",
        "certifi",
        "ipaddress; python_version=='2.7'",
    ], #  for works with https correctly
    "socks": ["PySocks"], # for works with SOCKS
}
What can this library, the main things, unlike built-in:
  • Thread safety.
  • Connection pooling.
  • Client-side SSL/TLS verification. (certifi)
  • File uploads with multipart encoding.
  • Helpers for retrying requests and dealing with HTTP redirects.
  • Support for gzip and deflate encoding.
  • Proxy support for HTTP and SOCKS.

requests

0.0.1 2011-02-13 (Frustration, Conception)

1.0.0 2012-12-17 (major version, production-ready)

2.23.0 2020-02-19 (latest release)

author Kenneth Reitz

Requests is an elegant and simple HTTP library for Python, built for human beings.

Requests uses:

  • chardet
  • idna
  • urllib3
  • certifi

Support versions:

  • 'Programming Language :: Python',
  • 'Programming Language :: Python :: 2',
  • 'Programming Language :: Python :: 2.7',
  • 'Programming Language :: Python :: 3',
  • 'Programming Language :: Python :: 3.5',
  • 'Programming Language :: Python :: 3.6',
  • 'Programming Language :: Python :: 3.7',
  • 'Programming Language :: Python :: 3.8',
  • 'Programming Language :: Python :: Implementation :: CPython',
  • 'Programming Language :: Python :: Implementation :: PyPy'
What can this library, the main things, unlike built-in:
  • Keep-Alive & Connection Pooling (from urllib3)
  • International Domains and URLs (from idna)
  • Sessions with Cookie Persistence (from urllib3)
  • Browser-style SSL Verification (from certifi)
  • Automatic Content Decoding (from urllib3)
  • Basic/Digest Authentication (from urllib3)
  • Elegant Key/Value Cookies (stdlib cookiejar)
  • Automatic Decompression (from urllib3)
  • Unicode Response Bodies
  • HTTP(S) Proxy Support (from urllib3)
  • Multipart File Uploads (from urllib3)
  • Streaming Downloads (from urllib3)
  • Connection Timeouts (from urllib3)
  • Chunked Requests (from urllib3)
  • .netrc Support (stdlib)
  • convenience (the main reason why the package has become popular)

async

non-stdlib

grequests

0.1.0 2012-05-11 (first commit)

0.4.0 2019-07-30 (latest release)

author Spencer Phillip Young

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

Note: You should probably use requests-threads or requests-futures instead.

grequests uses:

  • gevent
  • requests

Support versions:

  • 'Programming Language :: Python',
  • 'Programming Language :: Python :: 2.7',
  • 'Programming Language :: Python :: 3',
  • (as support latest gevent and requests)

requests-threads

0.1.0 2017-09-04 (first commit)

0.1.1 2017-09-04 (latest)

author Kenneth Reitz

A Requests session that returns awaitable Twisted Deferreds instead of response objects.

requests-threads uses:

  • requests
  • twisted

Support versions:

  • 'Programming Language :: Python',
  • 'Programming Language :: Python :: 2.6',
  • 'Programming Language :: Python :: 2.7',
  • 'Programming Language :: Python :: 3',
  • 'Programming Language :: Python :: 3.3',
  • 'Programming Language :: Python :: 3.4',
  • 'Programming Language :: Python :: 3.5',
  • 'Programming Language :: Python :: 3.6',
  • 'Programming Language :: Python :: Implementation :: CPython',
  • 'Programming Language :: Python :: Implementation :: PyPy'

requests-futures

0.9.0 2013-03-21 (initial import, working async requests via futures)

1.0.0 2019-06-11 (Bump version to 1.0.0) (major version, production-ready) (latest)

author Ross McFarland

Small add-on for the python requests http library. Makes use of python 3.2's concurrent.futures or the backport for prior versions of python. The additional API and changes are minimal and strives to avoid surprises.

requests-futures uses:

  • requests
  • futures if python_version<"3.2", else concurrent.futures in stdlib

Support versions:

  • 'Programming Language :: Python',
  • 'Programming Language :: Python :: 2.7',
  • 'Programming Language :: Python :: 3',
  • 'Programming Language :: Python :: 3.4',
  • 'Programming Language :: Python :: 3.5',
  • (and other, as support requests)

aiohttp (client side)

0.1 2013-10-25 (initial import)

1.0.0 2016-09-16 (Bump to 1.0) (major version, production-ready)

3.6.2 2019-10-09 (latest)

author Nikolay Kim (Andrew Svetlov is maintainer)

Asynchronous HTTP Client/Server for asyncio and Python.

aiohttp uses:

  • attrs
  • chardet
  • multidict
  • async_timeout
  • yarl
  • idna-ssl if python_version<"3.7"
  • typing_extensions

Support versions:

  • 'Programming Language :: Python',
  • 'Programming Language :: Python :: 3',
  • 'Programming Language :: Python :: 3.6',
  • 'Programming Language :: Python :: 3.7',
  • 'Programming Language :: Python :: 3.8',
What can this library, the main things, unlike built-in:
  • websockets
  • A broadly requests-compatible API.
  • convenience
  • Plus all the standard features of requests

httpx

0.0.1 2015-03-18 (initial commit)

0.12.0 2020-03-09 (latest)

A 1.0 release is expected to be issued sometime on or before April 2020.

author Tom Christie (also drf, mkdocs, uvicorn, starlette creator). https://www.encode.io/

HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.

httpx uses:

  • urllib3 - Sync client support.
  • h11 - HTTP/1.1 support.
  • h2 - HTTP/2 support.
  • certifi - SSL certificates.
  • chardet - Fallback auto-detection for response encoding.
  • hstspreload - determines whether IDNA-encoded host should be only accessed via HTTPS.
  • idna - Internationalized domain name support.
  • rfc3986 - URL parsing & normalization.
  • brotlipy - Decoding for "brotli" compressed responses. (Optional)

Support versions:

  • "Programming Language :: Python :: 3",
  • "Programming Language :: Python :: 3.6",
  • "Programming Language :: Python :: 3.7",
  • "Programming Language :: Python :: 3.8",
What can this library, the main things, unlike built-in:
  • A broadly requests-compatible API.
  • Standard synchronous interface, but with async support if you need it.
  • HTTP/1.1 and HTTP/2 support.
  • Ability to make requests directly to WSGI applications or ASGI applications.
  • Strict timeouts everywhere.
  • Fully type annotated.
  • 99% test coverage.
  • Plus all the standard features of requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment