Simple web server to simulate a read timout error implemented as a Bash script.
Using netcat + pv (Pipe Viewer) to simulate an slow PyPI custom index (almost LOTL :-p).
Bash simple file server with bandwidth limit simulated by pv.
First, download a real python package:
pip3.11 download --no-cache-dir lxml
# rename file
mv lxml-some-version.whl lxml-orig.whlStart the slow file server with 5kB/s max download speed
#!/bin/bash
sfile='lxml-orig.whl'
host='::1'
port='8081'
HTTP11_200='HTTP/1.1 200 OK'
content_length_header="Content-Length: $(wc -c < "$sfile")"
empty_end=''
max_bytes='150k'
rate_limit='5k'
pv_opts=( '--size' "$max_bytes" '--stop-at-size' '--rate-limit' "$rate_limit" "$sfile")
(
# send HTTP 1.1 message
printf "%s\r\n" "$HTTP11_200" "$content_length_header" "$empty_end"
# send 150k bytes worth of binary data at 5kB/s
pv ${pv_opts[*]} ) | \
# start a first server to send 150k bytes of data
netcat -l "$host" "$port" && \
# start a second server to avoid connect timeout error
netcat -l "$host" "$port"Output:
GET /lxml/ HTTP/1.104KiB/s] [============> ] 6% ETA 0:00:30
Host: [::1]:8081
<redacted>
150KiB 0:00:30 [4.99KiB/s] [==============================================================================================================================================================================================================================>] 100%
GET /pip/ HTTP/1.1
Host: [::1]:8081
<redacted>
Will fail twice due to Read time out. The first time will download 150k bytes and throw error:
Could not fetch URL http://[::1]:8081/lxml/: connection error: HTTPConnectionPool(host='::1', port=8081): Read timed out. - skipping
The second time will not read anything and throw the expected error that includes the timeout value:
Could not fetch URL http://[::1]:8081/pip/: connection error: HTTPConnectionPool(host='::1', port=8081): Max retries exceeded with url: /pip/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='::1', port=8081): Read timed out. (read timeout=5)")) - skipping
Download command:
time pip3.11 download -vv --timeout 5 --retries 0 --no-cache-dir --index-url=http://[::1]:8081 lxmlComplete Output:
Created temporary directory: /tmp/pip-build-tracker-50pehdq0
Initialized build tracking at /tmp/pip-build-tracker-50pehdq0
Created build tracker: /tmp/pip-build-tracker-50pehdq0
Entered build tracker: /tmp/pip-build-tracker-50pehdq0
Created temporary directory: /tmp/pip-download-npo08ab4
Looking in indexes: http://[::1]:8081
1 location(s) to search for versions of lxml:
* http://[::1]:8081/lxml/
Fetching project page and analyzing links: http://[::1]:8081/lxml/
Getting page http://[::1]:8081/lxml/
Found index url http://[::1]:8081/
Starting new HTTP connection (1): ::1:8081
http://::1:8081 "GET /lxml/ HTTP/1.1" 200 5013672
Could not fetch URL http://[::1]:8081/lxml/: connection error: HTTPConnectionPool(host='::1', port=8081): Read timed out. - skipping
Skipping link: not a file: http://[::1]:8081/lxml/
Given no hashes to check 0 links for project 'lxml': discarding no candidates
ERROR: Could not find a version that satisfies the requirement lxml (from versions: none)
1 location(s) to search for versions of pip:
* http://[::1]:8081/pip/
Fetching project page and analyzing links: http://[::1]:8081/pip/
Getting page http://[::1]:8081/pip/
Found index url http://[::1]:8081/
Starting new HTTP connection (1): ::1:8081
Could not fetch URL http://[::1]:8081/pip/: connection error: HTTPConnectionPool(host='::1', port=8081): Max retries exceeded with url: /pip/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='::1', port=8081): Read timed out. (read timeout=5)")) - skipping
Skipping link: not a file: http://[::1]:8081/pip/
Given no hashes to check 0 links for project 'pip': discarding no candidates
No remote pip version found
ERROR: No matching distribution found for lxml
Exception information:
Traceback (most recent call last):
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 174, in _add_to_criteria
raise RequirementsConflicted(criterion)
pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('lxml')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 399, in resolve
raise ResolutionImpossible(e.criterion.information)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('lxml'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
status = _inner_run()
^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
return self.run(options, args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/commands/download.py", line 132, in run
requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lmc/.local/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 104, in resolve
raise error from e
pip._internal.exceptions.DistributionNotFound: No matching distribution found for lxml
Removed build tracker: '/tmp/pip-build-tracker-50pehdq0'
real 0m37.438s
user 0m0.351s
sys 0m0.020s
https://medium.com/@zakharenko/how-to-simulate-network-failures-in-linux-b71ab585e86f