Skip to content

Instantly share code, notes, and snippets.

@dmtucker
Last active April 25, 2020 04:07
Show Gist options
  • Select an option

  • Save dmtucker/bffe536fabd27b5ebf43db112a89fa6c to your computer and use it in GitHub Desktop.

Select an option

Save dmtucker/bffe536fabd27b5ebf43db112a89fa6c to your computer and use it in GitHub Desktop.
What's up with Python 1.17 in the PyPI stats database?
@dmtucker
Copy link
Copy Markdown
Author

dmtucker commented Mar 5, 2018

Where does 1.17 come from?

Standard urllib seems like the most likely source (which is present in at least one fork):

$ python -c 'import sys, urllib, urllib2;  print(sys.version[:3], urllib.__version__, urllib2.__version__)'; python3 -c 'import urllib.request; print(urllib.request.__version__)'
('2.7', '1.17', '2.7')
3.5

What agents may be using urllib?

@dmtucker
Copy link
Copy Markdown
Author

dmtucker commented Mar 6, 2018

Where is 1.17 being used from?

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' country
Served from cache: False
Data processed: 81.80 GiB
Data billed: 81.80 GiB
Estimated cost: $0.40

| country | download_count |
| ------- | -------------- |
| US      |      3,216,292 |
| JP      |         55,160 |
| CN      |         34,118 |
| GB      |         24,925 |
| IN      |          8,064 |
| None    |          7,190 |
| FR      |          6,131 |
| DE      |          5,703 |
| IL      |          5,109 |
| ES      |          4,710 |

@dmtucker
Copy link
Copy Markdown
Author

dmtucker commented Mar 6, 2018

Notes

Switching PyPI to HTTPS-only should mean that occurrences dropped in Oct/Nov 2017 (but they didn't).

urllib.urlretrieve uses FancyURLopener which inherits from URLopener which sets the User-Agent.

easy_install's user-agent logic:
https://github.com/pypa/setuptools/blob/97ff22f31ace57f4eabb6f1e77c9c553de0d1c24/setuptools/package_index.py#L50

log parser user agent logic:
https://github.com/pypa/linehaul/blob/master/linehaul/user_agents.py


(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' system distro
Served from cache: False
Data processed: 146.60 GiB
Data billed: 146.60 GiB
Estimated cost: $0.72

| system_name | distro_name | download_count |
| ----------- | ----------- | -------------- |
| None        | None        |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' installer installer-version
Served from cache: False
Data processed: 153.04 GiB
Data billed: 153.04 GiB
Estimated cost: $0.75

| installer_name | installer_version | download_count |
| -------------- | ----------------- | -------------- |
| None           | None              |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' impl impl-version
Served from cache: False
Data processed: 155.91 GiB
Data billed: 155.91 GiB
Estimated cost: $0.77

| implementation | impl_version | download_count |
| -------------- | ------------ | -------------- |
| None           | None         |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' setuptools-version
Served from cache: False
Data processed: 51.03 GiB
Data billed: 51.03 GiB
Estimated cost: $0.25

| setuptools_version | download_count |
| ------------------ | -------------- |
| None               |      3,405,944 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 365 --where 'details.python = "1.17"' '' openssl
Served from cache: False
Data processed: 173.99 GiB
Data billed: 173.99 GiB
Estimated cost: $0.85

| openssl_version | download_count |
| --------------- | -------------- |
| None            |      3,405,944 |

Python Version Trends

None of these seem to correspond with 1.17 trends:

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "2.6"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        146,529 |
| 2018-02        |        828,609 |
| 2018-01        |      1,141,411 |
| 2017-12        |      1,614,495 |
| 2017-11        |      1,371,796 |
| 2017-10        |      2,023,681 |
| 2017-09        |      2,289,735 |
| 2017-08        |      2,704,477 |
| 2017-07        |      2,828,229 |
| 2017-06        |      2,889,239 |
| 2017-05        |      2,043,001 |
| 2017-04        |      2,457,703 |
| 2017-03        |      2,680,095 |
| 2017-02        |      2,576,469 |
| 2017-01        |      3,066,075 |
| 2016-12        |      3,053,485 |
| 2016-11        |      3,591,192 |
| 2016-10        |      3,270,814 |
| 2016-09        |      3,209,299 |
| 2016-08        |      4,005,600 |
| 2016-07        |      3,946,704 |
| 2016-06        |      4,171,922 |
| 2016-05        |      2,034,229 |
| 2016-03        |      1,815,018 |
| 2016-02        |      9,519,408 |
| 2016-01        |      2,959,021 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "2.7"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |      4,668,812 |
| 2018-02        |     22,073,881 |
| 2018-01        |     30,490,386 |
| 2017-12        |     31,818,101 |
| 2017-11        |     30,530,294 |
| 2017-10        |     42,417,489 |
| 2017-09        |     47,877,172 |
| 2017-08        |     61,447,951 |
| 2017-07        |     63,732,403 |
| 2017-06        |     66,186,931 |
| 2017-05        |     68,284,527 |
| 2017-04        |     71,634,346 |
| 2017-03        |     75,668,717 |
| 2017-02        |     72,209,408 |
| 2017-01        |     80,589,481 |
| 2016-12        |     71,611,590 |
| 2016-11        |     78,672,198 |
| 2016-10        |     68,610,169 |
| 2016-09        |     74,346,875 |
| 2016-08        |     72,597,214 |
| 2016-07        |     62,769,844 |
| 2016-06        |     50,753,857 |
| 2016-05        |     17,155,656 |
| 2016-03        |      9,538,368 |
| 2016-02        |     48,248,300 |
| 2016-01        |     15,114,515 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.4"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        453,018 |
| 2018-02        |      1,966,827 |
| 2018-01        |      2,435,945 |
| 2017-12        |      2,517,948 |
| 2017-11        |      2,310,739 |
| 2017-10        |      2,491,198 |
| 2017-09        |      2,332,760 |
| 2017-08        |      2,697,636 |
| 2017-07        |      2,397,228 |
| 2017-06        |      1,834,341 |
| 2017-05        |      1,863,789 |
| 2017-04        |      1,848,761 |
| 2017-03        |      2,238,398 |
| 2017-02        |      1,794,172 |
| 2017-01        |      1,793,060 |
| 2016-12        |      1,519,826 |
| 2016-11        |      1,539,866 |
| 2016-10        |      1,660,522 |
| 2016-09        |      1,602,485 |
| 2016-08        |      1,999,833 |
| 2016-07        |      1,976,705 |
| 2016-06        |        792,186 |
| 2016-05        |        363,550 |
| 2016-03        |        188,615 |
| 2016-02        |        977,270 |
| 2016-01        |        258,988 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.5"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        419,007 |
| 2018-02        |      2,124,505 |
| 2018-01        |      2,331,339 |
| 2017-12        |      2,649,399 |
| 2017-11        |      2,759,223 |
| 2017-10        |      2,312,185 |
| 2017-09        |      2,016,890 |
| 2017-08        |      2,438,073 |
| 2017-07        |      2,391,372 |
| 2017-06        |      3,384,902 |
| 2017-05        |      2,832,727 |
| 2017-04        |      3,037,491 |
| 2017-03        |      2,466,478 |
| 2017-02        |      2,367,826 |
| 2017-01        |      2,417,595 |
| 2016-12        |      2,464,993 |
| 2016-11        |      1,820,581 |
| 2016-10        |      2,006,853 |
| 2016-09        |      1,761,309 |
| 2016-08        |      1,242,226 |
| 2016-07        |      1,004,178 |
| 2016-06        |        603,266 |
| 2016-05        |        333,189 |
| 2016-03        |         89,432 |
| 2016-02        |        489,781 |
| 2016-01        |        105,938 |

(david-YYe3LIIa) david@kahuna:~ $ pypinfo --days 1095 --limit 36 --order download_month --where 'details.python = "3.6"' '' month
Served from cache: False
Data processed: 186.30 GiB
Data billed: 186.30 GiB
Estimated cost: $0.91

| download_month | download_count |
| -------------- | -------------- |
| 2018-03        |        460,217 |
| 2018-02        |      2,271,910 |
| 2018-01        |      2,236,416 |
| 2017-12        |      2,251,264 |
| 2017-11        |      5,606,128 |
| 2017-10        |      2,226,206 |
| 2017-09        |      2,054,285 |
| 2017-08        |      2,339,605 |
| 2017-07        |      1,661,571 |
| 2017-06        |      1,726,659 |
| 2017-05        |      1,152,307 |
| 2017-04        |        902,785 |
| 2017-03        |        792,730 |
| 2017-02        |        580,227 |
| 2017-01        |        391,527 |
| 2016-12        |         71,144 |
| 2016-11        |         26,199 |
| 2016-10        |         23,336 |
| 2016-09        |         23,020 |
| 2016-08        |         23,118 |
| 2016-07        |         20,472 |
| 2016-06        |         10,369 |
| 2016-05        |          3,076 |
| 2016-03        |          1,319 |
| 2016-02        |          7,425 |
| 2016-01        |          1,597 |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment