Skip to content

Instantly share code, notes, and snippets.

@netskink
Last active August 5, 2024 17:11
Show Gist options
  • Save netskink/abcfd95049207f6206cc8315a7e15d8c to your computer and use it in GitHub Desktop.
Save netskink/abcfd95049207f6206cc8315a7e15d8c to your computer and use it in GitHub Desktop.
six moves issue draft

Using python 3.12.1 on z/OS UNIX. Using a venv to install various libs. Google chromium depot tools uses python and six to get code from a git repo.

TODO: This was originally written for the six github issues tracker.

Setup

  • python 3.12.1
  • six 1.16.0 (six-1.16.0-py2.py3-none-any.whl.metadata)
  • z/OS

The explicit problem is that python fails to resolve six.moves.

Error message

Failed to fetch file gs://v8-wasm-spec-tests/3548183660d38fac0b5fc96171a19b1340ea902c for test/wasm-spec-tests/tests.tar.gz. [Err: Traceback (most recent call last):
File "/xxx/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gsutil", line 21, in <module>  
  gsutil.RunMain() 
File "/xxx/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gsutil.py", line 121, in RunMain
    import gslib.__main__
File "/xxx/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/__main__.py", line 36, in <module>
    from six.moves import configparser
ModuleNotFoundError: No module named 'six.moves'

Attempted solutions:

  • Force reinstall of six
  • Force install fresh and not cached
  • Attempt to use six.moves from urllib3
    • $ pip install urllib3
  • Remove ~/.cache
    • Removing ~/.cache was the only sure method for installing modules without using cached versions.
  • Modifying/Creating PYTHON_EXECUTABLE

Current workflow

Clean up to ensure clean environment

Ensure a pristine setup

$ rm -rf ~/.cached venv
xxx@yyy zzz  (main)
$ python3 --version
Python 3.12.1
xxx@yyy zzz  (main)

Create venv and install libs

No need to uses options on pip install since venv and .cached are removed.

$ python -m venv venv
xxx@yyy zzz  (main)
$ . venv/bin/activate
(venv) xxx@yyy zzz  (main)
$ python -m pip install -U pip
Requirement already satisfied: pip in ./venv/lib/python3.12/site-packages (23.3.2)
Collecting pip
  Downloading pip-24.2-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.2-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 2.8 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.3.2
    Uninstalling pip-23.3.2:
      Successfully uninstalled pip-23.3.2
Successfully installed pip-24.2
(venv) xxx@yyy zzz  (main)
$ pip install six
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0
(venv) xxx@yyy zzz  (main)

Usage and error

  • On z/OS UNIX, type is the equivalent of which on linux.

  • NOTE: This simplified process has six.modules import working. There is hope! Something must be amiss with depot_tools.

$ type python
python is hashed (/xxxx/venv/bin/python)
(venv) xxx@yyy zzz  (main)
$ python
Python 3.12.1 (heads/pyz_dev-3.12:fff9b52d2e, Feb 24 2024, 12:18:13) [Clang 14.0.0 (build 1465bdb)] on zos
Type "help", "copyright", "credits" or "license" for more information.
>>> from six.moves import configparser
>>>

Any advice is appreciated

Doing a trace of the path during the run, results in this path prior to fetch running.

PYTHON3: python is cached /xxxx/v8port.upstream/venv/bin/python
fetch3: fetch is /xxxx/zopen/usr/local/zopen/depot_tools/depot_tools-main/fetch
PATH 3: /xxxx/zopen/dev/v8port.upstream/venv/bin:/xxxx/zopen/usr/local/zopen/depot_tools/depot_tools-main:
/xxxx/zopen/usr/local/zopen/gn/gn-main.20240524_183021.zos:
/xxxx/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20240724_025947.zos/bin:
/xxxx/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20240724_025947.zos/altbin:
/xxxx/zopen/usr/local/zopen/sed/sed-4.9.20240719_144929.zos/bin:
/xxxx/zopen/usr/local/zopen/sed/sed-4.9.20240719_144929.zos/altbin:
/usr/lpp/IBM/cyp/v3r12/pyz/bin:
/xxxx/zopen/usr/local/zopen/ninja/ninja-heads.v1.12.1.20240717_020611.zos/bin:
/xxxx/zopen/usr/local/zopen/ninja/ninja-heads.v1.12.1.20240717_020611.zos/altbin:
/xxxx/zopen/usr/local/zopen/make/make-4.4.1.20240627_150632.zos/bin:
/xxxx/zopen/usr/local/zopen/make/make-4.4.1.20240627_150632.zos/altbin:
/xxxx/zopen/usr/local/zopen/jq/jq-1.6.20240104_213906.zos/bin:
/xxxx/zopen/usr/local/zopen/bash/bash-5.2.21.20240617_152629.zos/bin:
/xxxx/zopen/usr/local/zopen/perl/perl5-heads.v5.41.2.20240722_194015.zos/bin:
/xxxx/zopen/usr/local/zopen/perl/perl5-heads.v5.41.2.20240722_194015.zos/altbin:
/xxxx/zopen/usr/local/zopen/ncurses/ncurses-6.5.20240507_195319.zos/bin:
/xxxx/zopen/usr/local/zopen/less/less-heads.v661.20240720_035954.zos/bin:
/xxxx/zopen/usr/local/zopen/less/less-heads.v661.20240720_035954.zos/altbin:
/xxxx/zopen/usr/local/zopen/git/git-heads.v2.45.2.20240724_182931.zos/bin:
/xxxx/zopen/usr/local/zopen/git/git-heads.v2.45.2.20240724_182931.zos/altbin:
/xxxx/zopen/usr/local/zopen/curl/curl-8.9.0.20240724_132428.zos/bin:
/xxxx/zopen/usr/local/zopen/curl/curl-8.9.0.20240724_132428.zos/altbin:
.:
/C/CCplus/LangV2GA/usr/lpp/IBM/oelcpp/v2r0/bin:
/xxxx/zopen/usr/local/zopen/comp_clang/comp_clang-DEV.20240229_200034.zos/bin:
/xxxx/zopen/usr/local/zopen/cmake/CMake-heads.v3.27.8.20240129_142242.zos/bin:
/bin

Doing a find of six related files when installing urllib3 or six. It makes no difference which particular package is installed, the result for six related files are the same.

$ find . -name "six*"
./venv/lib/python3.12/site-packages/pip/_vendor/urllib3/packages/six.py
./venv/lib/python3.12/site-packages/pip/_vendor/urllib3/packages/__pycache__/six.cpython-312.pyc
./venv/lib/python3.12/site-packages/six.py
./venv/lib/python3.12/site-packages/six-1.16.0.dist-info
./venv/lib/python3.12/site-packages/__pycache__/six.cpython-312.pyc
./v8base/v8/test/test262/data/src/insignificant-input-elements/six-per-em-space.case
./v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/third_party/six
./v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/third_party/six/six.py
./v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/vendored/boto/boto/vendored/six.py

workflow

$ cd ~/zopen/dev/v8port.upstream
# This is currently v8port using branch of the pull requesrt
$ rm -rf ~/.local v8base install ${HOME}/zopen/usr/local/zopen/v8 venv log ~/.cache; zopen build > gabylogfile 2>&1

result

This is the result when build is attempted using modified pull request

$ tail -f gabylogfile
### zopen_build ###
export DEPOT_TOOLS_UPDATE=0
export DEPOT_TOOLS_BOOTSTRAP_PYTHON3=0
export VPYTHON_BYPASS='manually managed python not supported by chrome operations'
export CUSTOM_CIPD_CLIENT=/x/yyy/zopen/usr/local/zopen/depot_tools/depot_tools-main.20240726_201652.zos/.cipd_client
    Creating a python venv...
activating the virtual env - venv
updating the virtual env pip
Requirement already satisfied: pip in ./venv/lib/python3.12/site-packages (23.3.2)
Collecting pip
  Downloading pip-24.2-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.2-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 8.3 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.3.2
    Uninstalling pip-23.3.2:
      Successfully uninstalled pip-23.3.2
Successfully installed pip-24.2
PYTHON : python is cached /x/yyy/zopen/dev/v8port.upstream/venv/bin/python
fetch : fetch is /x/yyy/zopen/usr/local/zopen/depot_tools/depot_tools-main.20240726_201652.zos/fetch
PATH : /x/yyy/zopen/dev/v8port.upstream/venv/bin:/z/jd895801/zopen/usr/local/zopen/depot_tools/depot_tools-main.20240726_201652.zos:/z/jd895801/zopen/usr/local/zopen/gn/gn-main.20240524_183021.zos:/z/jd895801/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20240724_025947.zos/bin:/z/jd895801/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20240724_025947.zos/altbin:/z/jd895801/zopen/usr/local/zopen/sed/sed-4.9.20240719_144929.zos/bin:/z/jd895801/zopen/usr/local/zopen/sed/sed-4.9.20240719_144929.zos/altbin:/usr/lpp/IBM/cyp/v3r12/pyz/bin:/z/jd895801/zopen/usr/local/zopen/ninja/ninja-heads.v1.12.1.20240717_020611.zos/bin:/z/jd895801/zopen/usr/local/zopen/ninja/ninja-heads.v1.12.1.20240717_020611.zos/altbin:/z/jd895801/zopen/usr/local/zopen/make/make-4.4.1.20240627_150632.zos/bin:/z/jd895801/zopen/usr/local/zopen/make/make-4.4.1.20240627_150632.zos/altbin:/z/jd895801/zopen/usr/local/zopen/jq/jq-1.6.20240104_213906.zos/bin:/z/jd895801/zopen/usr/local/zopen/bash/bash-5.2.21.20240617_152629.zos/bin:/z/jd895801/zopen/usr/local/zopen/perl/perl5-heads.v5.41.2.20240722_194015.zos/bin:/z/jd895801/zopen/usr/local/zopen/perl/perl5-heads.v5.41.2.20240722_194015.zos/altbin:/z/jd895801/zopen/usr/local/zopen/ncurses/ncurses-6.5.20240507_195319.zos/bin:/z/jd895801/zopen/usr/local/zopen/less/less-heads.v661.20240720_035954.zos/bin:/z/jd895801/zopen/usr/local/zopen/less/less-heads.v661.20240720_035954.zos/altbin:/z/jd895801/zopen/usr/local/zopen/git/git-heads.v2.45.2.20240724_182931.zos/bin:/z/jd895801/zopen/usr/local/zopen/git/git-heads.v2.45.2.20240724_182931.zos/altbin:/z/jd895801/zopen/usr/local/zopen/curl/curl-8.9.0.20240724_132428.zos/bin:/z/jd895801/zopen/usr/local/zopen/curl/curl-8.9.0.20240724_132428.zos/altbin:.:/C/CCplus/LangV2GA/usr/lpp/IBM/oelcpp/v2r0/bin:/z/jd895801/zopen/usr/local/zopen/comp_clang/comp_clang-DEV.20240229_200034.zos/bin:/z/jd895801/zopen/usr/local/zopen/cmake/CMake-heads.v3.27.8.20240129_142242.zos/bin:/bin
DEPOT_TOOLS_HOME : /x/yyy/zopen/usr/local/zopen/depot_tools/depot_tools-main.20240726_201652.zos
CUSTOM_CIPD_CLIENT : /x/yyy/zopen/usr/local/zopen/depot_tools/depot_tools-main.20240726_201652.zos/.cipd_client
VPYTHON_BYPASS : manually managed python not supported by chrome operations
Installing reqired package in venv
Collecting httplib2 (from -r requirements.txt (line 1))
  Downloading httplib2-0.22.0-py3-none-any.whl.metadata (2.6 kB)
Collecting setuptools (from -r requirements.txt (line 4))
  Downloading setuptools-72.1.0-py3-none-any.whl.metadata (6.6 kB)
Collecting urllib3 (from -r requirements.txt (line 8))
  Downloading urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB)
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 (from httplib2->-r requirements.txt (line 1))
  Downloading pyparsing-3.1.2-py3-none-any.whl.metadata (5.1 kB)
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
Downloading setuptools-72.1.0-py3-none-any.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 4.5 MB/s eta 0:00:00
Downloading urllib3-2.2.2-py3-none-any.whl (121 kB)
Downloading pyparsing-3.1.2-py3-none-any.whl (103 kB)
Installing collected packages: urllib3, setuptools, pyparsing, httplib2
Successfully installed httplib2-0.22.0 pyparsing-3.1.2 setuptools-72.1.0 urllib3-2.2.2
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0
Invoking gclient metrics --opt-out
You're not a Googler. Metrics collection is disabled for you.
fetch v8...

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://chromium.googlesource.com/v8/v8.git /z/jd895801/zopen/dev/v8port.upstream/v8base/_gclient_v8_vu9t108h' in '/z/jd895801/zopen/dev/v8port.upstream/v8base'
Cloning into '/z/jd895801/zopen/dev/v8port.upstream/v8base/_gclient_v8_vu9t108h'...
remote: Sending approximately 851.07 MiB ...
remote: Counting objects: 7751, done
remote: Finding sources: 100% (295/295)
remote: Total 1120729 (delta 935237), reused 1120555 (delta 935237)
Receiving objects: 100% (1120729/1120729), 853.29 MiB | 19.44 MiB/s, done.
Resolving deltas: 100% (935237/935237), done.
[0:05:50] Still working on:
[0:05:50]   v8
[0:05:54] Still working on:
[0:05:54]   v8
[0:05:54] Still working on:
[0:05:54]   v8
________ running 'python3 third_party/depot_tools/update_depot_tools_toggle.py --disable' in '/x/yyy/zopen/dev/v8port.upstream/v8base/v8'
________ running 'python3 build/landmines.py --landmine-scripts tools/get_landmines.py' in '/x/yyy/zopen/dev/v8port.upstream/v8base/v8'
________ running 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth -u --bucket v8-wasm-spec-tests -s test/wasm-spec-tests/tests.tar.gz.sha1' in '/z/jd895801/zopen/dev/v8port.upstream/v8base/v8'
Failed to fetch file gs://v8-wasm-spec-tests/3548183660d38fac0b5fc96171a19b1340ea902c for test/wasm-spec-tests/tests.tar.gz. [Err: Traceback (most recent call last):
  File "/x/yyy/zopen/dev/v8port.upstream/v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gsutil", line 21, in <module>
    gsutil.RunMain()
  File "/x/yyy/zopen/dev/v8port.upstream/v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gsutil.py", line 121, in RunMain
    import gslib.__main__
  File "/x/yyy/zopen/dev/v8port.upstream/v8base/v8/third_party/depot_tools/external_bin/gsutil/gsutil_4.68/gsutil/gslib/__main__.py", line 36, in <module>
    from six.moves import configparser
ModuleNotFoundError: No module named 'six.moves'
]
...
stuff truncated
Hook 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth -u --bucket v8-wasm-spec-tests -s test/wasm-spec-tests/tests.tar.gz.sha1' took 313.55 secs
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "name": "v8",
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "deps_file": "DEPS",
    "managed": False,
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Subprocess failed with return code 2.
JFD EARLY EXIT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment