# step 0 - cleanup your existing drivers
sudo apt-get --purge remove "*nvidia*"
sudo apt-get --purge remove "*cuda*" "*cudnn*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" "*libnccl*"
# step 0.1 - disable iommu
ll /sys/class/iommu/
# if this folder is empty, continue
# if the folder is not empty, see https://docs.dolphinics.com/latest/guides/iommu.html
sudo reboot
# step 1 - install drivers
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
# Search for available Nvidia drivers and install the latest version.
apt search --names-only nvidia-driver
sudo apt install nvidia-driver-560-open # or latest
sudo reboot
# step 1.1 - verify driver
nvidia-smi
# ---
# step 2 - install cuda & cudnn
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
apt search --names-only cuda-toolkit
sudo apt install cuda-toolkit-12-6 # or latest
apt search --names-only cudnn
sudo apt install cudnn9-cuda-12-6 # or latest
sudo reboot
# step 2.1 - verify driver
nvidia-smi
# ---
# step 3 - make cuda samples
sudo apt install git cmake
# Clone the Nvidia CUDA samples repository to test CUDA installation.
cd ~
git clone https://github.com/nvidia/cuda-samples
cd cuda-samples
make -j `nproc`
# ---
# step 4 - test p2p
./bin/x86_64/linux/release/deviceQuery
# look for "> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : {Yes/No}"
# if you see Yes - stop, you already have p2p
# if you see No - continue
# ---
# step 5 - uninstall official driver (just the driver)
sudo apt remove nvidia-driver-550-open
sudo apt remove nvidia-dkms-550-open
sudo apt remove nvidia-driver-550-server-open
sudo apt remove nvidia-dkms-550-server-open
# step 5.1 - double check
sudo dpkg -l | grep nvidia
# verify that all nvidia drivers / dkms are uninstalled
# ---
# step 6 - install patched driver
cd ~
git clone [email protected]:tinygrad/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules
# step 6.1 - Building your driver
nvidia-smi # note down driver version - you've uninstalled the driver, but it should still be loaded in memory, if not, use `dpkg -l | grep nvidia` to grab the version `560.35.03-0ubuntu1` => `560.35.03`
git remote add upstream [email protected]:NVIDIA/open-gpu-kernel-modules.git
git fetch --all
# rebase patch with git or your preferred GUI
git rebase -Xignore-space-change -i upstream/560.35.03 # your driver version - make sure tag exists - you might have to deal with a conflicting README
# step 6.2 - make driver
./install.sh
# make modules -j$(nproc)
# sudo checkinstall make modules_install -j$(nproc)
# name = nvidia-driver-550-open-patch-tinygrad
# version = {driver-version}-p2p
sudo depmod
sudo reboot
# step 6.3 verify driver
nvidia-smi
# ---
# step 7 - test p2p
cd ~/cuda-samples
./bin/x86_64/linux/release/deviceQuery
# look for "> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : {Yes/No}"
# if you see Yes - stop, you're done
# if you see No - go to troubleshooting
# ---
# troubleshooting
see https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5
see https://morgangiraud.medium.com/multi-gpu-tinygrad-patch-4904a75f8e16
Last active
April 3, 2025 03:14
-
-
Save legraphista/c7f11c29dcc415a309406ae6da941e6e to your computer and use it in GitHub Desktop.
nvidia driver with p2p support for rtx 4090
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "!pip install torch plotly tqdm numpy matplotlib seaborn" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 8, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "import torch\n", | |
| "import gc\n", | |
| "import numpy as np" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "[device(type='cuda', index=0),\n", | |
| " device(type='cuda', index=1),\n", | |
| " device(type='cuda', index=2),\n", | |
| " device(type='cuda', index=3),\n", | |
| " device(type='cuda', index=4),\n", | |
| " device(type='cuda', index=5)]" | |
| ] | |
| }, | |
| "execution_count": 2, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "list_of_gpus = [torch.device(f\"cuda:{i}\") for i in range(torch.cuda.device_count())]\n", | |
| "list_of_gpus" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 3, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "cuda:0 -> cuda:3\n", | |
| "cuda:1 -> cuda:4\n", | |
| "cuda:5 -> cuda:3\n", | |
| "cuda:0 -> cuda:1\n", | |
| "cuda:4 -> cuda:5\n", | |
| "cuda:2 -> cuda:1\n", | |
| "cuda:4 -> cuda:0\n", | |
| "cuda:5 -> cuda:1\n", | |
| "cuda:3 -> cuda:0\n", | |
| "cuda:1 -> cuda:5\n", | |
| "cuda:3 -> cuda:2\n", | |
| "cuda:4 -> cuda:1\n", | |
| "cuda:5 -> cuda:0\n", | |
| "cuda:4 -> cuda:2\n", | |
| "cuda:0 -> cuda:5\n", | |
| "cuda:3 -> cuda:1\n", | |
| "cuda:2 -> cuda:0\n", | |
| "cuda:4 -> cuda:3\n", | |
| "cuda:1 -> cuda:0\n", | |
| "cuda:2 -> cuda:4\n", | |
| "cuda:3 -> cuda:5\n", | |
| "cuda:1 -> cuda:2\n", | |
| "cuda:0 -> cuda:4\n", | |
| "cuda:2 -> cuda:5\n", | |
| "cuda:3 -> cuda:4\n", | |
| "cuda:5 -> cuda:2\n", | |
| "cuda:1 -> cuda:3\n", | |
| "cuda:5 -> cuda:4\n", | |
| "cuda:2 -> cuda:3\n", | |
| "cuda:0 -> cuda:2\n", | |
| "All 30 GPU transfer combinations are present\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "speed_matrix = np.zeros((len(list_of_gpus), len(list_of_gpus)))\n", | |
| "\n", | |
| "test_pattern = []\n", | |
| "for gpu1 in list_of_gpus:\n", | |
| " for gpu2 in list_of_gpus:\n", | |
| " if gpu1 == gpu2:\n", | |
| " continue\n", | |
| " test_pattern.append((gpu1, gpu2))\n", | |
| "\n", | |
| "# Shuffle and ensure no adjacent pairs share GPUs\n", | |
| "np.random.shuffle(test_pattern)\n", | |
| "i = 0\n", | |
| "while i < len(test_pattern)-1:\n", | |
| " gpu1, gpu2 = test_pattern[i]\n", | |
| " next_gpu1, next_gpu2 = test_pattern[i+1]\n", | |
| " \n", | |
| " # If adjacent pairs share any GPU, swap with next non-conflicting pair\n", | |
| " if gpu1 in (next_gpu1, next_gpu2) or gpu2 in (next_gpu1, next_gpu2):\n", | |
| " for j in range(i+2, len(test_pattern)):\n", | |
| " candidate_gpu1, candidate_gpu2 = test_pattern[j]\n", | |
| " if gpu1 not in (candidate_gpu1, candidate_gpu2) and gpu2 not in (candidate_gpu1, candidate_gpu2):\n", | |
| " test_pattern[i+1], test_pattern[j] = test_pattern[j], test_pattern[i+1]\n", | |
| " break\n", | |
| " i += 1\n", | |
| "\n", | |
| "\n", | |
| "for gpu1, gpu2 in test_pattern:\n", | |
| " print(f\"{gpu1} -> {gpu2}\")\n", | |
| "\n", | |
| "# Verify all GPU combinations are present\n", | |
| "expected_combinations = set()\n", | |
| "for gpu1 in list_of_gpus:\n", | |
| " for gpu2 in list_of_gpus:\n", | |
| " if gpu1 != gpu2:\n", | |
| " expected_combinations.add((gpu1, gpu2))\n", | |
| "\n", | |
| "test_pattern_set = set(test_pattern)\n", | |
| "\n", | |
| "assert len(expected_combinations) == len(test_pattern_set), \"Some GPU combinations are missing\"\n", | |
| "assert expected_combinations == test_pattern_set, \"Test pattern doesn't match expected GPU combinations\"\n", | |
| "\n", | |
| "print(f\"All {len(test_pattern)} GPU transfer combinations are present\")\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "GPU cuda:0 can access: GPU cuda:1: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓, GPU cuda:4: ✓, GPU cuda:5: ✓, \n", | |
| "GPU cuda:1 can access: GPU cuda:0: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓, GPU cuda:4: ✓, GPU cuda:5: ✓, \n", | |
| "GPU cuda:2 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:3: ✓, GPU cuda:4: ✓, GPU cuda:5: ✓, \n", | |
| "GPU cuda:3 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:2: ✓, GPU cuda:4: ✓, GPU cuda:5: ✓, \n", | |
| "GPU cuda:4 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓, GPU cuda:5: ✓, \n", | |
| "GPU cuda:5 can access: GPU cuda:0: ✓, GPU cuda:1: ✓, GPU cuda:2: ✓, GPU cuda:3: ✓, GPU cuda:4: ✓, \n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "for gpu1 in list_of_gpus:\n", | |
| " print(f\"GPU {gpu1} can access: \", end=\"\")\n", | |
| " for gpu2 in list_of_gpus:\n", | |
| " if gpu1 == gpu2:\n", | |
| " continue\n", | |
| " can_access = torch.cuda.can_device_access_peer(gpu1, gpu2)\n", | |
| " print(f\"GPU {gpu2}: {'✓' if can_access else '✗'}, \", end=\" \")\n", | |
| " print()\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 5, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 0%| | 0/30 [00:00<?, ?it/s]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:0 to cuda:3\n", | |
| " Generating random tensor for cuda:0\n", | |
| " Copying to cuda:0\n", | |
| " Copying from cuda:0 to cuda:3\n", | |
| " Done in 0.08 seconds at 26.16 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 3%|▎ | 1/30 [00:04<02:22, 4.92s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:1 to cuda:4\n", | |
| " Generating random tensor for cuda:1\n", | |
| " Copying to cuda:1\n", | |
| " Copying from cuda:1 to cuda:4\n", | |
| " Done in 0.08 seconds at 26.15 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 7%|▋ | 2/30 [00:09<02:20, 5.00s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:5 to cuda:3\n", | |
| " Generating random tensor for cuda:5\n", | |
| " Copying to cuda:5\n", | |
| " Copying from cuda:5 to cuda:3\n", | |
| " Done in 0.08 seconds at 26.12 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 10%|█ | 3/30 [00:14<02:13, 4.95s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:0 to cuda:1\n", | |
| " Generating random tensor for cuda:0\n", | |
| " Copying to cuda:0\n", | |
| " Copying from cuda:0 to cuda:1\n", | |
| " Done in 0.08 seconds at 26.16 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 13%|█▎ | 4/30 [00:19<02:06, 4.88s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:4 to cuda:5\n", | |
| " Generating random tensor for cuda:4\n", | |
| " Copying to cuda:4\n", | |
| " Copying from cuda:4 to cuda:5\n", | |
| " Done in 0.08 seconds at 26.13 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 17%|█▋ | 5/30 [00:24<02:01, 4.86s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:2 to cuda:1\n", | |
| " Generating random tensor for cuda:2\n", | |
| " Copying to cuda:2\n", | |
| " Copying from cuda:2 to cuda:1\n", | |
| " Done in 0.08 seconds at 26.11 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 20%|██ | 6/30 [00:29<01:59, 4.97s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:4 to cuda:0\n", | |
| " Generating random tensor for cuda:4\n", | |
| " Copying to cuda:4\n", | |
| " Copying from cuda:4 to cuda:0\n", | |
| " Done in 0.08 seconds at 26.15 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 23%|██▎ | 7/30 [00:34<01:55, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:5 to cuda:1\n", | |
| " Generating random tensor for cuda:5\n", | |
| " Copying to cuda:5\n", | |
| " Copying from cuda:5 to cuda:1\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 27%|██▋ | 8/30 [00:39<01:50, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:3 to cuda:0\n", | |
| " Generating random tensor for cuda:3\n", | |
| " Copying to cuda:3\n", | |
| " Copying from cuda:3 to cuda:0\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 30%|███ | 9/30 [00:44<01:45, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:1 to cuda:5\n", | |
| " Generating random tensor for cuda:1\n", | |
| " Copying to cuda:1\n", | |
| " Copying from cuda:1 to cuda:5\n", | |
| " Done in 0.08 seconds at 26.12 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 33%|███▎ | 10/30 [00:49<01:40, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:3 to cuda:2\n", | |
| " Generating random tensor for cuda:3\n", | |
| " Copying to cuda:3\n", | |
| " Copying from cuda:3 to cuda:2\n", | |
| " Done in 0.08 seconds at 26.15 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 37%|███▋ | 11/30 [00:54<01:35, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:4 to cuda:1\n", | |
| " Generating random tensor for cuda:4\n", | |
| " Copying to cuda:4\n", | |
| " Copying from cuda:4 to cuda:1\n", | |
| " Done in 0.08 seconds at 26.04 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 40%|████ | 12/30 [00:59<01:30, 5.01s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:5 to cuda:0\n", | |
| " Generating random tensor for cuda:5\n", | |
| " Copying to cuda:5\n", | |
| " Copying from cuda:5 to cuda:0\n", | |
| " Done in 0.08 seconds at 26.06 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 43%|████▎ | 13/30 [01:04<01:25, 5.00s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:4 to cuda:2\n", | |
| " Generating random tensor for cuda:4\n", | |
| " Copying to cuda:4\n", | |
| " Copying from cuda:4 to cuda:2\n", | |
| " Done in 0.08 seconds at 26.13 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 47%|████▋ | 14/30 [01:09<01:20, 5.00s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:0 to cuda:5\n", | |
| " Generating random tensor for cuda:0\n", | |
| " Copying to cuda:0\n", | |
| " Copying from cuda:0 to cuda:5\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 50%|█████ | 15/30 [01:14<01:15, 5.00s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:3 to cuda:1\n", | |
| " Generating random tensor for cuda:3\n", | |
| " Copying to cuda:3\n", | |
| " Copying from cuda:3 to cuda:1\n", | |
| " Done in 0.08 seconds at 26.01 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 53%|█████▎ | 16/30 [01:19<01:09, 4.99s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:2 to cuda:0\n", | |
| " Generating random tensor for cuda:2\n", | |
| " Copying to cuda:2\n", | |
| " Copying from cuda:2 to cuda:0\n", | |
| " Done in 0.08 seconds at 26.04 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 57%|█████▋ | 17/30 [01:24<01:04, 4.99s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:4 to cuda:3\n", | |
| " Generating random tensor for cuda:4\n", | |
| " Copying to cuda:4\n", | |
| " Copying from cuda:4 to cuda:3\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 60%|██████ | 18/30 [01:29<00:59, 4.98s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:1 to cuda:0\n", | |
| " Generating random tensor for cuda:1\n", | |
| " Copying to cuda:1\n", | |
| " Copying from cuda:1 to cuda:0\n", | |
| " Done in 0.08 seconds at 25.94 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 63%|██████▎ | 19/30 [01:34<00:54, 4.99s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:2 to cuda:4\n", | |
| " Generating random tensor for cuda:2\n", | |
| " Copying to cuda:2\n", | |
| " Copying from cuda:2 to cuda:4\n", | |
| " Done in 0.08 seconds at 26.12 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 67%|██████▋ | 20/30 [01:39<00:49, 4.99s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:3 to cuda:5\n", | |
| " Generating random tensor for cuda:3\n", | |
| " Copying to cuda:3\n", | |
| " Copying from cuda:3 to cuda:5\n", | |
| " Done in 0.08 seconds at 25.99 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 70%|███████ | 21/30 [01:44<00:45, 5.02s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:1 to cuda:2\n", | |
| " Generating random tensor for cuda:1\n", | |
| " Copying to cuda:1\n", | |
| " Copying from cuda:1 to cuda:2\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 73%|███████▎ | 22/30 [01:49<00:40, 5.02s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:0 to cuda:4\n", | |
| " Generating random tensor for cuda:0\n", | |
| " Copying to cuda:0\n", | |
| " Copying from cuda:0 to cuda:4\n", | |
| " Done in 0.08 seconds at 26.08 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 77%|███████▋ | 23/30 [01:54<00:35, 5.03s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:2 to cuda:5\n", | |
| " Generating random tensor for cuda:2\n", | |
| " Copying to cuda:2\n", | |
| " Copying from cuda:2 to cuda:5\n", | |
| " Done in 0.08 seconds at 26.01 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 80%|████████ | 24/30 [01:59<00:30, 5.04s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:3 to cuda:4\n", | |
| " Generating random tensor for cuda:3\n", | |
| " Copying to cuda:3\n", | |
| " Copying from cuda:3 to cuda:4\n", | |
| " Done in 0.08 seconds at 26.04 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 83%|████████▎ | 25/30 [02:04<00:25, 5.03s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:5 to cuda:2\n", | |
| " Generating random tensor for cuda:5\n", | |
| " Copying to cuda:5\n", | |
| " Copying from cuda:5 to cuda:2\n", | |
| " Done in 0.08 seconds at 26.04 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 87%|████████▋ | 26/30 [02:09<00:20, 5.03s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:1 to cuda:3\n", | |
| " Generating random tensor for cuda:1\n", | |
| " Copying to cuda:1\n", | |
| " Copying from cuda:1 to cuda:3\n", | |
| " Done in 0.08 seconds at 25.97 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 90%|█████████ | 27/30 [02:15<00:15, 5.04s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:5 to cuda:4\n", | |
| " Generating random tensor for cuda:5\n", | |
| " Copying to cuda:5\n", | |
| " Copying from cuda:5 to cuda:4\n", | |
| " Done in 0.08 seconds at 25.94 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 93%|█████████▎| 28/30 [02:20<00:10, 5.03s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:2 to cuda:3\n", | |
| " Generating random tensor for cuda:2\n", | |
| " Copying to cuda:2\n", | |
| " Copying from cuda:2 to cuda:3\n", | |
| " Done in 0.09 seconds at 25.11 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| " 97%|█████████▋| 29/30 [02:25<00:05, 5.02s/it]" | |
| ] | |
| }, | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Testing cuda:0 to cuda:2\n", | |
| " Generating random tensor for cuda:0\n", | |
| " Copying to cuda:0\n", | |
| " Copying from cuda:0 to cuda:2\n", | |
| " Done in 0.08 seconds at 25.92 GB/s\n" | |
| ] | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "100%|██████████| 30/30 [02:30<00:00, 5.00s/it]\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "import time\n", | |
| "from tqdm import tqdm\n", | |
| "\n", | |
| "\n", | |
| "for gpu1, gpu2 in tqdm(test_pattern):\n", | |
| " print(f\"Testing {gpu1} to {gpu2}\")\n", | |
| "\n", | |
| " gc.collect()\n", | |
| " torch.cuda.empty_cache()\n", | |
| "\n", | |
| " # Warm up transfer\n", | |
| " warm_up = torch.randn(1024, 1024, device=gpu1)\n", | |
| " _ = warm_up.to(gpu2)\n", | |
| " torch.cuda.synchronize(gpu1)\n", | |
| " torch.cuda.synchronize(gpu2)\n", | |
| " del warm_up\n", | |
| "\n", | |
| " # 8gb\n", | |
| " print(f\" Generating random tensor for {gpu1}\")\n", | |
| " random_tensor = torch.tensor(np.random.randint(0, 100, (1024, 1024, 1024), dtype=np.int16))\n", | |
| "\n", | |
| " # copy from CPU to GPU 1\n", | |
| " print(f\" Copying to {gpu1}\")\n", | |
| " gpu1_tensor = random_tensor.to(gpu1)\n", | |
| "\n", | |
| " # copy from GPU 1 to GPU 2\n", | |
| " _start = time.perf_counter()\n", | |
| " print(f\" Copying from {gpu1} to {gpu2}\")\n", | |
| " gpu2_tensor = gpu1_tensor.to(gpu2)\n", | |
| " torch.cuda.synchronize(gpu1)\n", | |
| " torch.cuda.synchronize(gpu2)\n", | |
| " _end = time.perf_counter()\n", | |
| "\n", | |
| " _total_time = _end - _start\n", | |
| " _bytes_moved = random_tensor.numel() * random_tensor.element_size()\n", | |
| " _speed_GBs = _bytes_moved / _total_time / 1e9\n", | |
| " print(f\" Done in {_end - _start:.2f} seconds at {_speed_GBs:.2f} GB/s\")\n", | |
| "\n", | |
| " speed_matrix[gpu1.index, gpu2.index] = _speed_GBs\n", | |
| "\n", | |
| " # pull tensor back to CPU\n", | |
| " cpu_tensor = gpu2_tensor.to(\"cpu\")\n", | |
| "\n", | |
| " # check if the tensors are equal\n", | |
| " if not torch.equal(random_tensor, cpu_tensor):\n", | |
| " print(f\"GPU tensors are not equal\")\n", | |
| "\n", | |
| " del gpu2_tensor\n", | |
| " del gpu1_tensor\n", | |
| " del random_tensor\n", | |
| "\n", | |
| "gc.collect()\n", | |
| "torch.cuda.empty_cache()\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 9, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAJOCAYAAABLKeTiAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAa0JJREFUeJzt3XlYVeXax/HfZp5BQQETBUUxp3IuLTWnTHtNmzxmGR5TO2KZ5kRZDpVo07HUbDpqpTbbMe1k5mypmQqZphTOs2YCjiDs9f7hcZ92DLJhD2z8fq5rXZes4Vn3ekK7ubn3s0yGYRgCAAAA3IyHqwMAAAAASoNEFgAAAG6JRBYAAABuiUQWAAAAbolEFgAAAG6JRBYAAABuiUQWAAAAbolEFgAAAG6JRBYAAABuiUQWAFxk6dKluvHGG+Xn5yeTyaTMzExXh1RuxcbGKjEx0dVhAChnSGSBUti7d6+GDh2qunXrKiAgQAEBAapfv76SkpK0bds2q3MnTJggk8lk2a6cO27cOGVnZxc47/fffy/0ng0bNlT79u2vGtvkyZP173//uyyPV6ScnBxNnz5dt9xyiypVqiQfHx9Vq1ZNPXr00Icffqj8/HzLufv27bN6bk9PT9WoUUO9evVSWlpagfNefvnlQu/58ssvy2Qyad++fYUe/+t9ituKGsMVTp06pfvvv1/+/v6aOXOmPvjgAwUGBjr0nvv27VP//v1Vu3Zt+fn5KSoqSm3bttX48eMdel8AcBQvVwcAuJslS5aod+/e8vLyUt++fXXDDTfIw8NDu3bt0sKFCzVr1izt3btXNWvWtLpu1qxZCgoK0tmzZ7Vs2TK98MILWrlypb7//nuZTCa7xTd58mTde++96tmzp93GlKSTJ0/qjjvu0JYtW3T77bdr3Lhxqly5so4dO6bly5frgQceUEZGhp555hmr6/r06aNu3bopPz9fO3fu1KxZs/T1119r48aNuvHGG8scV5UqVfTBBx9Y7XvllVd06NAh/fOf/yxwbnnx448/6syZM3ruuefUqVMnh98vIyNDLVq0kL+/v/7+978rNjZWR48e1datWzV16lRNnDjR4TEAgL2RyAI22L17t/72t7+pZs2aWrFihaKjo62OT506VW+88YY8PAr+suPee+9VRESEJOnRRx/VPffco4ULF2rjxo26+eabnRJ/WTz00ENKTU3V559/rrvvvtvqWHJysjZv3qz09PQC1zVt2lQPPvig5es2bdqoR48emjVrlt56660yxxUYGGg1viR99NFHOn36dIH9f2YYhi5evCh/f/8yx1AaJ06ckCSFhYXZbcxz584VWdX95z//qbNnzyotLa3AD1lXYgEAd0NrAWCDF198UefOndOcOXMKJLGS5OXlpccff1wxMTFXHatDhw6SLrcp2IvJZNK5c+f03nvvWX6d/ue+wtTUVN1xxx0KCQlRUFCQOnbsqI0bN1513A0bNuibb77RoEGDCiSxVzRv3lx9+/a96liOeO6SiI2N1Z133qlvvvlGzZs3l7+/vyWRnjNnjjp06KCqVavK19dX9evX16xZs4oc47vvvlPLli3l5+enWrVq6f3337c679KlS5o4caLq1KkjPz8/hYeH65ZbbtG3334rSWrfvr0efvhhSVKLFi0K/Hf64Ycf1LVrV4WGhiogIEDt2rXT999/b3WPK60ov/zyix544AFVqlRJt9xyS5HPv3v3blWvXr1AEitJVatWLfQ5ly1bZunhrV+/vhYuXFjg2szMTD3xxBOKiYmRr6+v4uPjNXXqVJnNZqvzzGazpk2bpgYNGsjPz0+RkZEaPHiwTp8+bXWeYRh6/vnnVb16dQUEBOi2227Tjh07Ctz3anMM4NpARRawwZIlSxQfH69WrVqVeazdu3dLksLDw8s81hUffPCBHnnkEbVs2VKDBg2SJNWuXVuStGPHDt16660KCQnR6NGj5e3trbfeekvt27fXmjVrin2mxYsXS1KxFc6ScsRzl1R6err69OmjwYMHa+DAgUpISJB0ue2jQYMG6tGjh7y8vLR48WINGTJEZrNZSUlJVmNkZGTo3nvv1YABA/Twww9r9uzZSkxMVLNmzdSgQQNJl5PMlJQUy3+L7Oxsbd68WVu3blXnzp319NNPKyEhQW+//bYmTZqkuLg4y3+nlStX6o477lCzZs00fvx4eXh4WBLtdevWqWXLllbx3HfffapTp44mT54swzCKfPaaNWtq+fLlWrlypeWHieL89ttv6t27tx599FE9/PDDmjNnju677z4tXbpUnTt3liSdP39e7dq10+HDhzV48GDVqFFD69evV3Jyso4ePapp06ZZxhs8eLDmzp2r/v376/HHH9fevXs1Y8YMpaam6vvvv5e3t7ck6dlnn9Xzzz+vbt26qVu3btq6dau6dOmi3Nxcq/iuNscArhEGgBLJysoyJBk9e/YscOz06dPGyZMnLdv58+ctx8aPH29IMtLT042TJ08ae/fuNd566y3D19fXiIyMNM6dO2d13smTJwu9f4MGDYx27dpdNc7AwEDj4YcfLrC/Z8+eho+Pj7F7927LviNHjhjBwcFG27Ztix2zV69ehiQjMzPTav+FCxesnvv06dOWY3v37jUkGRMnTjROnjxpHDt2zFi9erXRpEkTQ5Lx+eefW5330ksvFXrvl156yZBk7N2796rPfkX37t2NmjVrWu2rWbOmIclYunRpgfP//N/rittvv92oVatWoWOsXbvWsu/EiROGr6+v8eSTT1r23XDDDUb37t2LjXHOnDmGJOPHH3+07DObzUadOnWM22+/3TCbzVbxxcXFGZ07d7bsu/L90qdPn2Lvc8X27dsNf39/Q5Jx4403GsOGDTP+/e9/W77/CnvOK/+NDOPy9390dLTRpEkTy77nnnvOCAwMNH799Ver68eOHWt4enoaBw4cMAzDMNatW2dIMubPn2913tKlS632nzhxwvDx8TG6d+9u9fxPPfWUIcnq+7okcwyg4qO1ACihKysMBAUFFTjWvn17ValSxbLNnDmzwDkJCQmqUqWK4uLiNHjwYMXHx+urr75SQECAw2PPz8/XsmXL1LNnT9WqVcuyPzo6Wg888IC+++47qxUU/qqoZ3/zzTetnruwX22PHz9eVapUUVRUlNq3b6/du3dr6tSpRbYoOFJcXJxuv/32Avv/3CeblZWl33//Xe3atdOePXuUlZVldW79+vV16623Wr6uUqWKEhIStGfPHsu+sLAw7dixQ7/99ptN8aWlpem3337TAw88oFOnTun333/X77//rnPnzqljx45au3ZtgV/ZP/rooyUau0GDBkpLS9ODDz6offv26bXXXlPPnj0VGRmpd955p8D51apVU69evSxfh4SEqF+/fkpNTdWxY8ckSZ9++qluvfVWVapUyRLr77//rk6dOik/P19r1661nBcaGqrOnTtbndesWTMFBQVp1apVkqTly5crNzdXjz32mNUHIJ944okC8ZV2jgFULLQWACUUHBwsSTp79myBY2+99ZbOnDmj48ePF/nr988//1whISHy9vZW9erVLb9KtkVpVzc4efKkzp8/b/lV+p9df/31MpvNOnjwoOVX43/152cPDQ217L/nnnvUsGFDSdKTTz5ptfzWFYMGDdJ9990nDw8PhYWFqUGDBvL19bX5GeyxskNcXFyh+7///nuNHz9eGzZs0Pnz562OZWVlWT1zjRo1ClxfqVIlq17PSZMm6a677lLdunXVsGFDde3aVQ899JAaN25cbHxXkrIr/bOFycrKUqVKla76TIWpW7euPvjgA+Xn5+uXX37RkiVL9OKLL2rQoEGKi4uzWj0hPj6+wJzXrVtX0uVlvKKiovTbb79p27ZtRa4GceVDZL/99puysrIK9OL+9bz9+/dLkurUqWN1vEqVKlbPLJV+jgFULCSyQAmFhoYqOjpa27dvL3DsSn9pceuUtm3b1rJqQWH8/PwkSRcuXCj0+Pnz5y3nOFu9evUkSdu3b1ebNm0s+2NiYiwfbLtSlfurOnXqFLu8VEme+8/nlUVhKxTs3r1bHTt2VL169fTqq68qJiZGPj4++s9//qN//vOfBSqgnp6ehY5t/Kk/tW3bttq9e7cWLVqkZcuW6d1339U///lPvfnmm3rkkUeKjO/KvV566aUilyb7a1W8NKsueHp6qlGjRmrUqJFuvvlm3XbbbZo/f77Ny4CZzWZ17txZo0ePLvT4lcTXbDaratWqmj9/fqHnlWZZtNLOMYCKhUQWsEH37t317rvvatOmTQU+dFNWVz5Nnp6eXmDVg/Pnz+vgwYPq0qXLVccprHJZpUoVBQQEFLo81q5du+Th4VHsSgt33nmnpkyZovnz51slsvZQXGzS5fkICAgo9oeAsli8eLFycnL05ZdfWlVbr/y6u7QqV66s/v37q3///jp79qzatm2rCRMmFJtkXanSh4SEOGVtWenyahOSdPToUav9GRkZMgzD6vvp119/lXR5VQPpcrxnz569aqy1a9fW8uXL1aZNm2IT7yt/B3777TerFpiTJ08WWN1AKt0cA6hY6JEFbDB69GgFBATo73//u44fP17guFHMp8avpmPHjvLx8dGsWbMKVAHffvtt5eXl6Y477rjqOIGBgQVederp6akuXbpo0aJFVlXj48ePa8GCBbrlllsUEhJS5Jht2rRR586d9fbbb2vRokWFnlPaZ78S2+LFi3XgwAGrYwcOHNDixYvVpUuXIiuhZXVl3D/Hn5WVpTlz5pR6zFOnTll9HRQUpPj4eOXk5BR7XbNmzVS7dm29/PLLhbawnDx5stQxrVu3TpcuXSqw/z//+Y8kFWg7OXLkiL744gvL19nZ2Xr//fd14403KioqSpJ0//33W5Zm+6vMzEzl5eVZzsvPz9dzzz1X4Ly8vDzL92unTp3k7e2t6dOnW/33+PPqB1eUdo4BVCxUZAEb1KlTRwsWLFCfPn2UkJBgebOXYRjau3evFixYIA8PD1WvXt3msatWrapnn31W48aNU9u2bdWjRw8FBARo/fr1+vDDD9WlSxf93//931XHadasmZYvX65XX31V1apVU1xcnFq1aqXnn39e3377rW655RYNGTJEXl5eeuutt5STk6MXX3zxquPOmzdPXbt2Vc+ePXXHHXeoU6dOqlSpkuXNXmvXri1Rol2YyZMn66abblLTpk01aNAgxcbGat++fXr77bdlMpk0efLkUo1bEl26dJGPj4/+7//+T4MHD9bZs2f1zjvvqGrVqgWqlCVVv359tW/fXs2aNVPlypW1efNmffbZZxo6dGix13l4eOjdd9/VHXfcoQYNGqh///667rrrdPjwYa1atUohISGWpdBsNXXqVG3ZskV33323pY9069atev/991W5cuUCH6iqW7euBgwYoB9//FGRkZGaPXu2jh8/bpXgjxo1Sl9++aXuvPNOyxJk586d088//6zPPvtM+/btU0REhNq1a6fBgwcrJSVFaWlp6tKli7y9vfXbb7/p008/1WuvvaZ7771XVapU0ciRI5WSkqI777xT3bp1U2pqqr7++usCFfnSzjGACsaFKyYAbisjI8P4xz/+YcTHxxt+fn6Gv7+/Ua9ePePRRx810tLSrM692rJafzVv3jzjpptuMgIDAw1fX1+jXr16xsSJE42LFy+W6Ppdu3YZbdu2tSy19Ocli7Zu3WrcfvvtRlBQkBEQEGDcdtttxvr160v83BcuXDCmTZtm3HzzzUZISIjh5eVlREVFGXfeeacxf/58Iy8vz3Lu1ZbV+qudO3cavXv3NqpWrWp4eXkZVatWNf72t78ZO3fuLHF8VxS1/FZRyzV9+eWXRuPGjQ0/Pz8jNjbWmDp1qjF79uwCy34VNUa7du2slkZ7/vnnjZYtWxphYWGW740XXnjByM3NtZxT2PJbV6Smphp33323ER4ebvj6+ho1a9Y07r//fmPFihWWc2z9vvr++++NpKQko2HDhkZoaKjh7e1t1KhRw0hMTLRaku3Pz/nNN98YjRs3tnwffvrppwXGPXPmjJGcnGzEx8cbPj4+RkREhNG6dWvj5ZdftnpewzCMt99+22jWrJnh7+9vBAcHG40aNTJGjx5tHDlyxHJOfn6+MXHiRCM6Otrw9/c32rdvb2zfvt2oWbOm1fdySeYYQMVnMowy/C4UAFDhxMbGqmHDhlqyZImrQwGAYtEjCwAAALdEIgsAAAC3RCILAAAAt0SPLAAAANwSFVkAAAC4JRJZAAAAuCVeiGAHZrNZR44cUXBwcKGvBwUAAPZjGIbOnDmjatWqycOj/NTkLl68qNzcXIeN7+PjIz8/P4eN745IZO3gyJEjxb6nHgAA2N/BgwdL9SZFR7h48aLiagbp2Il8h90jKipKe/fuJZn9ExJZOwgODpZ0+S9Uce+rR9EuHbvB1SG4rRFHWrk6BLf22nU/ujoEt5aa47jq07XA22R2dQhu6dxZs7redMzy/9/yIDc3V8dO5Gv/lliFBNu/Spx9xqyazfYpNzeXRPZPSGTt4Eo7QUhICIlsKV06V35+NeRufIJ8XB2CW3PE/3CuJUE+zF9ZeNONViblsZ0vKNikoGD7x2VW+XvW8oB/gQAAAOCWqMgCAADYSb5hVr4DVujPN2hDKQwVWQAAALglKrIAAAB2YpYhs+xfknXEmBUBFVkAAAC4JSqyAAAAdmKWWY7oZnXMqO6PiiwAAADcEhVZAAAAO8k3DOUb9u9ndcSYFQEVWQAAALglKrIAAAB2wqoFzkUiCwAAYCdmGconkXUaWgsAAADglqjIAgAA2AmtBc5FRRYAAABuiYosAACAnbD8lnNRkQUAAIBboiILAABgJ+b/bo4YFwVRkQUAAIBboiILAABgJ/kOWkfWEWNWBFRkAQAA4JaoyAIAANhJvnF5c8S4KIiKLAAAANwSFVkAAAA7YdUC5yKRBQAAsBOzTMqXySHjoiBaCwAAAOCWqMgCAADYidm4vDliXBRERRYAAABuiYosAACAneQ7qEfWEWNWBFRkAQAA4JaoyAIAANgJFVnnoiILAAAAt0RFFgAAwE7MhklmwwHryDpgzIqAiiwAAADckssT2WPHjmnYsGGKj4+Xn5+fIiMj1aZNG82aNUvnz5+3nBcbGyuTySSTyaTAwEA1bdpUn376qeV4YmKievbsWWD81atXy2QyKTMzs8gY/vjjD/Xt21chISEKCwvTgAEDdPbsWXs+JgAAuAZc6ZF1xIaCXJrI7tmzR02aNNGyZcs0efJkpaamasOGDRo9erSWLFmi5cuXW50/adIkHT16VKmpqWrRooV69+6t9evXlzmOvn37aseOHfr222+1ZMkSrV27VoMGDSrzuAAAAHAcl/bIDhkyRF5eXtq8ebMCAwMt+2vVqqW77rpLhmH9Govg4GBFRUUpKipKM2fO1Lx587R48WK1bt261DHs3LlTS5cu1Y8//qjmzZtLkqZPn65u3brp5ZdfVrVq1Uo9NgAAuLbky0P5DqgT5tt9xIrBZRXZU6dOadmyZUpKSrJKYv/MZCq6jO7l5SVvb2/l5uaWKY4NGzYoLCzMksRKUqdOneTh4aEffvih0GtycnKUnZ1ttQEAAMC5XJbIZmRkyDAMJSQkWO2PiIhQUFCQgoKCNGbMmEKvzc3NVUpKirKystShQ4cyxXHs2DFVrVrVap+Xl5cqV66sY8eOFXpNSkqKQkNDLVtMTEyZYgAAABWD8d9VC+y9GaxaUCiXf9jrrzZt2qS0tDQ1aNBAOTk5VsfGjBmjoKAgBQQEaOrUqZoyZYq6d+/u9BiTk5OVlZVl2Q4ePOj0GAAAQPnDh72cy2U9svHx8TKZTEpPT7faX6tWLUmSv79/gWtGjRqlxMREBQUFKTIy0qr1ICQkRPv37y9wTWZmpjw9PYtsX4iKitKJEyes9uXl5emPP/5QVFRUodf4+vrK19e3+AcEAACAQ7msIhseHq7OnTtrxowZOnfuXImuiYiIUHx8vKKiogr0zyYkJGjHjh0Fqrhbt25VXFycvL29Cx3z5ptvVmZmprZs2WLZt3LlSpnNZrVq1crGpwIAANeyfMPDYRsKcumsvPHGG8rLy1Pz5s318ccfa+fOnUpPT9e8efO0a9cueXp6lnisvn37ymQyqV+/ftqyZYsyMjI0e/ZsTZs2TU8++WSR111//fXq2rWrBg4cqE2bNun777/X0KFD9be//Y0VCwAAAMoxly6/Vbt2baWmpmry5MlKTk7WoUOH5Ovrq/r162vkyJEaMmRIiccKCwvTunXrNHbsWPXo0UNZWVmKj4/Xq6++qgEDBhR77fz58zV06FB17NhRHh4euueee/T666+X9fEAAMA1xiyTzA6oE5plXP2ka5DJ+OtirbBZdna2QkNDlZWVpZCQEFeH45YuHa3t6hDc1pDDbVwdglt7q/oGV4fg1rbklG0JxGudt8ns6hDc0tkzZt3a8Ei5+v/ulVzgq221FBhc8t8ol9S5M/nq3nhPuXrm8sClFVkAAICKxFErDLBqQeHoHAYAAIBboiILAABgJ45aYSCfTtBCUZEFAACAW6IiCwAAYCeXVy2wfz+rI8asCKjIAgAAwC1RkQUAALATszyUzzqyTkNFFgAAAG6JiiwAAICdsGqBc5HIAgAA2IlZHryi1oloLQAAAIBboiILAABgJ/mGSfmGA15R64AxKwIqsgAAAHBLVGQBAADsJN9By2/l0yNbKCqyAAAAcEtUZAEAAOzEbHjI7IDlt8wsv1UoKrIAAABwS1RkAQAA7IQeWeeiIgsAAAC3REUWAADATsxyzJqvZruPWDFQkQUAAIBboiILAABgJ2Z5yOyAOqEjxqwISGQBAADsJN/wUL4Dlt9yxJgVAbMCAABQgaSkpKhFixYKDg5W1apV1bNnT6Wnpxc4b8OGDerQoYMCAwMVEhKitm3b6sKFC8WOPXPmTMXGxsrPz0+tWrXSpk2bHPUYJUIiCwAAYCdmmRy2ldSaNWuUlJSkjRs36ttvv9WlS5fUpUsXnTt3znLOhg0b1LVrV3Xp0kWbNm3Sjz/+qKFDh8rDo+jU8OOPP9aIESM0fvx4bd26VTfccINuv/12nThxokxzVha0FgAAAFQgS5cutfp67ty5qlq1qrZs2aK2bdtKkoYPH67HH39cY8eOtZyXkJBQ7LivvvqqBg4cqP79+0uS3nzzTX311VeaPXu21TjOREUWAADATq70yDpik6Ts7GyrLScn56oxZWVlSZIqV64sSTpx4oR++OEHVa1aVa1bt1ZkZKTatWun7777rsgxcnNztWXLFnXq1Mmyz8PDQ506ddKGDRvKMmVlQiILAADgJmJiYhQaGmrZUlJSij3fbDbriSeeUJs2bdSwYUNJ0p49eyRJEyZM0MCBA7V06VI1bdpUHTt21G+//VboOL///rvy8/MVGRlptT8yMlLHjh2zw5OVDq0FAAAAduK4V9ReHvPgwYMKCQmx7Pf19S32uqSkJG3fvt2q2mo2X369wuDBgy1tAk2aNNGKFSs0e/bsqybH5QmJLAAAgJsICQmxSmSLM3ToUC1ZskRr165V9erVLfujo6MlSfXr17c6//rrr9eBAwcKHSsiIkKenp46fvy41f7jx48rKirKlkewK1oLAAAA7MRsmBy2lZRhGBo6dKi++OILrVy5UnFxcVbHY2NjVa1atQJLcv3666+qWbNmoWP6+PioWbNmWrFixf+e1WzWihUrdPPNN9swQ/ZFRRYAAKACSUpK0oIFC7Ro0SIFBwdbelhDQ0Pl7+8vk8mkUaNGafz48brhhht044036r333tOuXbv02WefWcbp2LGjevXqpaFDh0qSRowYoYcffljNmzdXy5YtNW3aNJ07d87SnuAKJLIAAAB2YnZQj6wtr6idNWuWJKl9+/ZW++fMmaPExERJ0hNPPKGLFy9q+PDh+uOPP3TDDTfo22+/Ve3atS3n7969W7///rvl6969e+vkyZN69tlndezYMd14441aunRpgQ+AOZPJMAzDZXevILKzsxUaGqqkdT3lG+Tt6nDc0is3fuLqENxW2oEYV4fg1r4528DVIbi1Jyr96uoQ3Nrx/OLfooTCnTljVsP6J5SVlVXiflFHu5ILTPmxnfyC7F8nvHg2T2NbrClXz1weUJEFAACwE7PhIbPhgIqsA8asCJgVAAAAuCUqsgAAAHaSL5PyVfIVBmwZFwWRyAIAANgJrQXOxawAAADALVGRBQAAsJN8OaYNIN/uI1YMVGQBAADglqjIAgAA2Ak9ss7FrAAAAMAtUZEFAACwk3zDQ/kOqJ46YsyKgFkBAACAW6IiCwAAYCeGTDI7YNUCgxciFIqKLAAAANwSFVkAAAA7oUfWuZgVAAAAuCUqsgAAAHZiNkwyG/bvZ3XEmBUBFVkAAAC4JSqyAAAAdpIvD+U7oE7oiDErAhJZAAAAO6G1wLlI7wEAAOCWqMgCAADYiVkeMjugTuiIMSsCZgUAAABuiYosAACAneQbJuU7oJ/VEWNWBFRkAQAA4JaoyAIAANgJqxY4FxVZAAAAuCUqsgAAAHZiGB4yG/avExoOGLMiYFYAAADglqjIAgAA2Em+TMqXA1YtcMCYFQEVWQAAALglKrIAAAB2YjYcs8KA2bD7kBUCiSwAAICdmB30YS9HjFkRMCsAAABwS1RkAQAA7MQsk8wO+GCWI8asCFxekT127JiGDRum+Ph4+fn5KTIyUm3atNGsWbN0/vx5y3mxsbEymUwymUwKDAxU06ZN9emnn1qOJyYmqmfPngXGX716tUwmkzIzM4uM4YUXXlDr1q0VEBCgsLAwOz4dAAAAHMWlieyePXvUpEkTLVu2TJMnT1Zqaqo2bNig0aNHa8mSJVq+fLnV+ZMmTdLRo0eVmpqqFi1aqHfv3lq/fn2Z48jNzdV9992nf/zjH2UeCwAAXLvyDZPDNhTk0taCIUOGyMvLS5s3b1ZgYKBlf61atXTXXXfJMKw/ohccHKyoqChFRUVp5syZmjdvnhYvXqzWrVuXKY6JEydKkubOnVumcQAAAOA8LktkT506ZanE/jmJ/TOTqeifPry8vOTt7a3c3FxHhQgAAGATVi1wLpfNSkZGhgzDUEJCgtX+iIgIBQUFKSgoSGPGjCn02tzcXKWkpCgrK0sdOnRwRrhWcnJylJ2dbbUBAADAucpder9p0yalpaWpQYMGysnJsTo2ZswYBQUFKSAgQFOnTtWUKVPUvXt3p8eYkpKi0NBQyxYTE+P0GAAAQPljlklmwwEbqxYUymWtBfHx8TKZTEpPT7faX6tWLUmSv79/gWtGjRqlxMREBQUFKTIy0qr1ICQkRPv37y9wTWZmpjw9PYtsXyiN5ORkjRgxwvJ1dnY2ySwAAICTuawiGx4ers6dO2vGjBk6d+5cia6JiIhQfHy8oqKiCvTPJiQkaMeOHQWquFu3blVcXJy8vb3tFruvr69CQkKsNgAAAOO/68jaezOoyBbKpa0Fb7zxhvLy8tS8eXN9/PHH2rlzp9LT0zVv3jzt2rVLnp6eJR6rb9++MplM6tevn7Zs2aKMjAzNnj1b06ZN05NPPlnstQcOHFBaWpoOHDig/Px8paWlKS0tTWfPni3rIwIAAMBBXLr8Vu3atZWamqrJkycrOTlZhw4dkq+vr+rXr6+RI0dqyJAhJR4rLCxM69at09ixY9WjRw9lZWUpPj5er776qgYMGFDstc8++6zee+89y9dNmjSRJK1atUrt27cv1bMBAIBrz5WeVkeMi4Jc/ora6OhoTZ8+XdOnTy/2vH379l11rLp162rhwoU2xzB37lzWkAUAAHAzLk9kAQAAKgrWkXUuElkAAAA7obXAuUjvAQAA4JaoyAIAANjJleWyHDEuCqIiCwAAALdERRYAAMBO6JF1LiqyAAAAcEtUZAEAAOyEiqxzUZEFAACAW6IiCwAAYCdUZJ2LiiwAAADcEhVZAAAAO6Ei61xUZAEAAOCWqMgCAADYiSHHvIXLsPuIFQMVWQAAALglKrIAAAB2Qo+sc5HIAgAA2AmJrHPRWgAAAAC3REUWAADATqjIOhcVWQAAALglKrIAAAB2QkXWuajIAgAAwC1RkQUAALATwzDJcED11BFjVgRUZAEAAOCWqMgCAADYiVkmh7yi1hFjVgRUZAEAAOCWqMgCAADYCasWOBcVWQAAALglKrIAAAB2wqoFzkUiCwAAYCe0FjgXrQUAAABwS1RkAQAA7ITWAueiIgsAAAC3REUWAADATgwH9chSkS0ciawd3RqcroBgT1eH4Zb2HYp2dQhu68YaR10dglsb5XGfq0Nwaw8euODqENzayvOxrg7BLV24kCfphKvDQDlAawEAAICdGJIMwwGbDTGkpKSoRYsWCg4OVtWqVdWzZ0+lp6dbndO+fXuZTCar7dFHHy123LNnz2ro0KGqXr26/P39Vb9+fb355pu2T5IdkcgCAABUIGvWrFFSUpI2btyob7/9VpcuXVKXLl107tw5q/MGDhyoo0ePWrYXX3yx2HFHjBihpUuXat68edq5c6eeeOIJDR06VF9++aUjH6dYtBYAAADYiVkmmeSAdWRtGHPp0qVWX8+dO1dVq1bVli1b1LZtW8v+gIAARUVFlXjc9evX6+GHH1b79u0lSYMGDdJbb72lTZs2qUePHiUex56oyAIAALiJ7Oxsqy0nJ+eq12RlZUmSKleubLV//vz5ioiIUMOGDZWcnKzz588XO07r1q315Zdf6vDhwzIMQ6tWrdKvv/6qLl26lP6ByoiKLAAAgJ04eh3ZmJgYq/3jx4/XhAkTirzObDbriSeeUJs2bdSwYUPL/gceeEA1a9ZUtWrVtG3bNo0ZM0bp6elauHBhkWNNnz5dgwYNUvXq1eXl5SUPDw+98847VlVeZyORBQAAcBMHDx5USEiI5WtfX99iz09KStL27dv13XffWe0fNGiQ5c+NGjVSdHS0OnbsqN27d6t27dqFjjV9+nRt3LhRX375pWrWrKm1a9cqKSlJ1apVU6dOncrwVKVHIgsAAGAnZsMkkwMqslfWpg0JCbFKZIszdOhQLVmyRGvXrlX16tWLPbdVq1aSpIyMjEIT2QsXLuipp57SF198oe7du0uSGjdurLS0NL388ssksgAAACg7wzD02GOP6YsvvtDq1asVFxd31WvS0tIkSdHRha/rfunSJV26dEkeHtYfr/L09JTZbC5zzKVFIgsAAGAnV9Z9dcS4JZWUlKQFCxZo0aJFCg4O1rFjxyRJoaGh8vf31+7du7VgwQJ169ZN4eHh2rZtm4YPH662bduqcePGlnHq1aunlJQU9erVSyEhIWrXrp1GjRolf39/1axZU2vWrNH777+vV1991d6PW2IksgAAAHbi6A97lcSsWbMkybJM1hVz5sxRYmKifHx8tHz5ck2bNk3nzp1TTEyM7rnnHo0bN87q/PT0dMuKB5L00UcfKTk5WX379tUff/yhmjVr6oUXXrjqixQciUQWAACgAjGuUr6NiYnRmjVrbB4nKipKc+bMKVNs9kYiCwAAYCfloSJ7LeGFCAAAAHBLVGQBAADsxNHLb8EaFVkAAAC4JSqyAAAAdlIelt+6llCRBQAAgFuiIgsAAGAnlyuyjli1wO5DVghUZAEAAOCWqMgCAADYCevIOhcVWQAAALglKrIAAAB2Yvx3c8S4KIhEFgAAwE5oLXAuWgsAAADglqjIAgAA2Au9BU5FRRYAAABuiYosAACAvTioR1b0yBaKiiwAAADcEhVZAAAAO7n8ilrHjIuCqMgCAADALVGRBQAAsBPWkXUuKrIAAABwS1RkAQAA7MUwOWaFASqyhaIiCwAAALfk8kT22LFjGjZsmOLj4+Xn56fIyEi1adNGs2bN0vnz5y3nxcbGymQyyWQyKTAwUE2bNtWnn35qOZ6YmKiePXsWGH/16tUymUzKzMws9P779u3TgAEDFBcXJ39/f9WuXVvjx49Xbm6uvR8VAABUcFdWLXDEhoJc2lqwZ88etWnTRmFhYZo8ebIaNWokX19f/fzzz3r77bd13XXXqUePHpbzJ02apIEDByo7O1uvvPKKevfureuuu06tW7cudQy7du2S2WzWW2+9pfj4eG3fvl0DBw7UuXPn9PLLL9vjMQEAAOAALk1khwwZIi8vL23evFmBgYGW/bVq1dJdd90l4y8/fgQHBysqKkpRUVGaOXOm5s2bp8WLF5cpke3atau6du1qde/09HTNmjWLRBYAANjG+O/miHFRgMsS2VOnTmnZsmWaPHmyVRL7ZyZT0Y3NXl5e8vb2dkgLQFZWlipXrmz3cQEAQMXG8lvO5bIe2YyMDBmGoYSEBKv9ERERCgoKUlBQkMaMGVPotbm5uUpJSVFWVpY6dOhg97imT5+uwYMHF3lOTk6OsrOzrTYAAAA4V4krskUla4GBgfL09LRbQJs2bZLZbFbfvn2Vk5NjdWzMmDEaN26cLl68qKCgIE2ZMkXdu3e3270PHz6srl276r777tPAgQOLPC8lJUUTJ060230BAEAFQhuA05S4IhsWFqZKlSoV2Pz9/ZWQkKB33nnHphvHx8fLZDIpPT3dan+tWrUUHx8vf3//AteMGjVKaWlpOnTokE6fPm1VsQ0JCVFWVlaBazIzM+Xp6Vlk+8IVR44c0W233abWrVvr7bffLvbc5ORkZWVlWbaDBw8Wez4AAADsr8QV2VWrVhW6PzMzU1u2bNGoUaPk5eWl/v37l2i88PBwde7cWTNmzNBjjz121URTutx2EB8fX+ixhIQEffTRR8rJyZGvr69l/9atWxUXFydvb+8ixz18+LBuu+02NWvWTHPmzJGHR/H5va+vr9U9AAAAJHpkna3EiWy7du2KPHbXXXcpNjZW06dPL3EiK0lvvPGG2rRpo+bNm2vChAlq3LixPDw89OOPP2rXrl1q1qxZicfq27evJk2apH79+mn06NEKDQ3V2rVrNW3aNL344otFXnf48GG1b99eNWvW1Msvv6yTJ09ajkVFRZX4/gAAAHAuu61a0K5dOz3xxBM2XVO7dm2lpqZq8uTJSk5O1qFDh+Tr66v69etr5MiRGjJkSInHCgsL07p16zR27Fj16NFDWVlZio+P16uvvqoBAwYUed23336rjIwMZWRkqHr16lbH/rr8FwAAQLFYfsup7JbIZmVlKTQ01ObroqOjNX36dE2fPr3Y8/bt23fVserWrauFCxfadP/ExEQlJibadA0AAABczy6J7KVLl/TSSy+pVatW9hgOAADATZn+uzliXPxViRPZu+++u9D9WVlZ2rFjh0wmk9atW2e3wAAAAIDilDiRLaptICYmRvfcc4/69u1bqtYCAACACoMeWacqcSI7Z84cR8YBAAAA2MSmHtmNGzdq8eLFys3NVceOHdW1a1dHxQUAAOB+qMg6VYkT2c8++0y9e/eWv7+/vL299eqrr2rq1KkaOXKkI+MDAAAAClXiV9SmpKRo4MCBysrK0unTp/X8889r8uTJjowNAADAvRgmx20ooMSJbHp6ukaOHClPT09J0pNPPqkzZ87oxIkTDgsOAADAnRiG4zYUVOJE9vz58woJCbF87ePjIz8/P509e9YhgQEAAADFsenDXu+++66CgoIsX+fl5Wnu3LmKiIiw7Hv88cftFx0AAIA74cNeTlXiRLZGjRp65513rPZFRUXpgw8+sHxtMplIZAEAAOAUJU5k9+3b58AwAAAAKgBHfTCLD3sVqsQ9sgAAAEB5UuKK7IULF7RixQrdeeedkqTk5GTl5ORYjnt6euq5556Tn5+f/aMEAABwAybj8uaIcVFQiRPZ9957T1999ZUlkZ0xY4YaNGggf39/SdKuXbtUrVo1DR8+3DGRAgAAAH9S4taC+fPna9CgQVb7FixYoFWrVmnVqlV66aWX9Mknn9g9QAAAALdhOHBDASVOZDMyMtSoUSPL135+fvLw+N/lLVu21C+//GLf6AAAAIAilLi1IDMz06on9uTJk1bHzWaz1XEAAIBrDqsWOFWJK7LVq1fX9u3bizy+bds2Va9e3S5BAQAAAFdT4kS2W7duevbZZ3Xx4sUCxy5cuKCJEyeqe/fudg0OAADArdAj61Qlbi146qmn9MknnyghIUFDhw5V3bp1JUnp6emaMWOG8vLy9NRTTzksUAAAgHKPV9QW68KFCzIMQwEBAZKk/fv364svvlD9+vXVpUsXm8crcSIbGRmp9evX6x//+IfGjh0rw7g8oyaTSZ07d9Ybb7yhyMhImwMAAADAteGuu+7S3XffrUcffVSZmZlq1aqVvL299fvvv+vVV1/VP/7xD5vGs+nNXnFxcVq6dKlOnjypjRs3auPGjTp58qSWLl2qWrVq2XRjAACACofWgmJt3bpVt956qyTps88+U2RkpPbv36/3339fr7/+us3jlbgi+2eVK1dWy5YtS3MpAAAArlHnz59XcHCwJGnZsmW6++675eHhoZtuukn79++3eTybKrIAAAAoxpXltxyxVQDx8fH697//rYMHD+qbb76x9MWeOHFCISEhNo9HIgsAAACnePbZZzVy5EjFxsaqVatWuvnmmyVdrs42adLE5vFK1VoAAACAgkzG5c0R41YE9957r2655RYdPXpUN9xwg2V/x44d1atXL5vHI5EFAACAQ9WoUUM9evRQjx491KFDB0VFRVkdL+1nr0rVWvDBBx+oTZs2qlatmqUxd9q0aVq0aFGpggAAAKgQWLWgUB988IF8fX2VlJSkiIgI9e7dW/Pnz1dmZmaZxrU5kZ01a5ZGjBihbt26KTMzU/n5+ZKksLAwTZs2rUzBAAAAoOJp166dXnnlFf3222/6/vvvdeONN2r69OmKiopShw4dNG3aNO3Zs8fmcW1OZKdPn6533nlHTz/9tDw9PS37mzdvrp9//tnmAAAAAHDtaNCggZKTk7Vx40bt3btXffr00YoVK9SwYUM1bNhQX331VYnHsrlHdu/evYV+qszX11fnzp2zdTgAAABco6KjozVw4EANHDhQ58+f1zfffCNfX98SX29zIhsXF6e0tDTVrFnTav/SpUt1/fXX2zocAABAhWGSg1YtsP+QLmcYhlatWqULFy6odevWqlSpks0rF9icyI4YMUJJSUm6ePGiDMPQpk2b9OGHHyolJUXvvvuurcNVKFU9zyjIk6V5S2PleV5xXFp9jzJ3ZfGt2faeLPzP4j2NXB2CW8vKD3R1CG7pYn6eq0OAjTIzMzVs2DBt3bpVN910k1555RV169ZN69evlyRVrVpVy5YtU+PGjW0a1+ZE9pFHHpG/v7/GjRun8+fP64EHHlC1atX02muv6W9/+5utwwEAAFQcjnoLl5u/2WvkyJHasGGDHn74YS1evFhdu3aVYRjasGGDPDw8NHr0aD399NNavHixTeOWah3Zvn37qm/fvjp//rzOnj2rqlWrlmYYAACAisVRS2W5+fJbX3/9tRYsWKB27dopMTFRMTExWrlypVq1aiVJmjp1qnr06GHzuKX6sFdeXp7q1KmjgIAABQQESJJ+++03eXt7KzY21uYgAAAAUHEdP35cdevWlSRdd9118vPzU0xMjOV4jRo1dPLkSZvHtbmhMzEx0dLP8Gc//PCDEhMTbQ4AAACgwuCFCIUym81Wy7Z6enrKZPpfu8Sf/2wLmyuyqampatOmTYH9N910k4YOHVqqIAAAAFCxvfvuuwoKCpIk5eXlae7cuYqIiJAknTlzplRj2pzImkymQm+WlZVlecsXAADAtchkOGj5LTevyNaoUUPvvPOO5euoqCh98MEHBc6xlc2JbNu2bZWSkqIPP/zQUiLOz89XSkqKbrnlFpsDAAAAQMW2b98+h4xrcyI7ZcoUtWvXTgkJCbr11lslSevWrVN2drZWrlxp9wABAADcBqsWFOrixYtavny57rzzTklScnKycnJyLMe9vLw0adIk+fn52TSuzR/2atCggbZt26b7779fJ06c0JkzZ9SvXz/t2rVLDRs2tHU4AAAAVHBz587VW2+9Zfl6xowZWr9+vVJTU5WamqoPPvhAb7zxhs3j2lSRvXTpkrp27ao333xTkydPtvlmAAAAFRoV2ULNnz9fo0ePttq3YMEC1ap1+e2U8+bN08yZMzVixAibxrWpIuvt7a1t27bZdAMAAABc2zIyMtSo0f9eae3n5ycPj/+loS1bttQvv/xi87g2txY8+OCD+te//mXzjQAAACq6K6sWOGJzZ5mZmVY9sSdPnrR6iZbZbLY6XlI2J7J5eXmaNWuWmjdvrsGDB2vEiBFWGwAAAFwnJSVFLVq0UHBwsKpWraqePXsqPT3d6pz27dvLZDJZbY8++uhVx965c6d69Oih0NBQBQYGqkWLFjpw4MBVr6tevbq2b99e5PFt27apevXqV3+4v7B51YLt27eradOmkqRff/3V6lhp38oAAABQIRimy5sjxi2hNWvWKCkpSS1atFBeXp6eeuopdenSRb/88osCAwMt5w0cOFCTJk2yfB0QEFDsuLt379Ytt9yiAQMGaOLEiQoJCdGOHTtKtNJAt27d9Oyzz6p79+4Fzr9w4YImTpyo7t27l/gZr7A5kV21apXNNwEAAIBzLF261OrruXPnqmrVqtqyZYvatm1r2R8QEKCoqKgSj/v000+rW7duevHFFy37ateuXaJrn3rqKX3yySdKSEjQ0KFDVbduXUlSenq6ZsyYYUm4bWVzawEAAACKYDhwk5SdnW21laSvNCsrS5JUuXJlq/3z589XRESEGjZsqOTkZJ0/f77IMcxms7766ivVrVtXt99+u6pWrapWrVrp3//+dwkmRYqMjNT69et1/fXXa+zYserVq5d69eql5ORk1a9fX999950iIyNLNNaf2VyRve2224ptIeClCAAA4Frl6FfUxsTEWO0fP368JkyYUOR1ZrNZTzzxhNq0aWO13v8DDzygmjVrqlq1atq2bZvGjBmj9PR0LVy4sNBxTpw4obNnz2rKlCl6/vnnNXXqVC1dulR33323Vq1apXbt2l31GeLi4rR06VL98ccfysjIkCTFx8cXSLBtYXMie+ONN1p9fenSJaWlpWn79u16+OGHSx0IAAAAinfw4EGFhIRYvvb19S32/KSkJG3fvl3fffed1f5BgwZZ/tyoUSNFR0erY8eO2r17d6HtAmazWZJ01113afjw4ZIu54Tr16/Xm2++WaJE9orKlSurZcuWJT6/ODYnsv/85z8L3T9hwgSdPXu2zAEBAAC4LQe/ECEkJMQqkS3O0KFDtWTJEq1du/aqKwK0atVK0uX1XgtLZCMiIuTl5aX69etb7b/++usLJMnOZLce2QcffFCzZ8+213AAAAAoBcMwNHToUH3xxRdauXKl4uLirnpNWlqaJCk6OrrQ4z4+PmrRokWBZbx+/fVX1axZs8wxl5bNFdmibNiwoUTLLwAAAFRYjnp5gQ1jJiUlacGCBVq0aJGCg4N17NgxSVJoaKj8/f21e/duLViwQN26dVN4eLi2bdum4cOHq23btmrcuLFlnHr16iklJUW9evWSJI0aNUq9e/dW27Ztddttt2np0qVavHixVq9ebc8ntYnNiezdd99t9bVhGDp69Kg2b96sZ555xm6BAQAAwHazZs2SdPmlB382Z84cJSYmysfHR8uXL9e0adN07tw5xcTE6J577tG4ceOszk9PT7eseCBJvXr10ptvvqmUlBQ9/vjjSkhI0Oeff65bbrnF4c9UFJsT2dDQUKuvPTw8lJCQoEmTJqlLly52CwwAAMDtOLhHtkSnGsWfHBMTozVr1pRqnL///e/6+9//XvJgHMzmRHbOnDmOiAMAAACwSal7ZLds2aKdO3dKkho0aKAmTZrYLSgAAAC3VA4qstcSmxPZEydO6G9/+5tWr16tsLAwSVJmZqZuu+02ffTRR6pSpYq9YwQAAAAKsHn5rccee0xnzpzRjh079Mcff+iPP/7Q9u3blZ2drccff9wRMQIAALiFK2/2csSGgmyuyC5dulTLly/X9ddfb9lXv359zZw5kw97AQAAwGlsrsiazWZ5e3sX2O/t7W15fRkAAADgaDYnsh06dNCwYcN05MgRy77Dhw9r+PDh6tixo12DAwAAcCuGAzcUYHMiO2PGDGVnZys2Nla1a9dW7dq1FRcXp+zsbE2fPt0RMQIAAAAF2NwjGxMTo61bt2r58uXatWuXJOn6669Xp06d7B4cAACAO3HUB7P4sFfhSrWOrMlkUufOndW5c2d7xwMAAACUSIlbCzZs2KAlS5ZY7Xv//fcVFxenqlWratCgQcrJybF7gAAAAG6F/linKXEiO2nSJO3YscPy9c8//6wBAwaoU6dOGjt2rBYvXqyUlBSHBAkAAAD8VYkT2bS0NKtVCT766CO1atVK77zzjkaMGKHXX39dn3zyic0BHDt2TMOGDVN8fLz8/PwUGRmpNm3aaNasWTp//rzlvNjYWJlMJplMJgUGBqpp06b69NNPLccTExPVs2fPAuOvXr1aJpNJmZmZRcbQo0cP1ahRQ35+foqOjtZDDz1ktSoDAABAibBqgVOVOJE9ffq0IiMjLV+vWbNGd9xxh+XrFi1a6ODBgzbdfM+ePWrSpImWLVumyZMnKzU1VRs2bNDo0aO1ZMkSLV++3Or8SZMm6ejRo0pNTVWLFi3Uu3dvrV+/3qZ7Fua2227TJ598ovT0dH3++efavXu37r333jKPCwAAAMcp8Ye9IiMjtXfvXsXExCg3N1dbt27VxIkTLcfPnDlT6IsSijNkyBB5eXlp8+bNCgwMtOyvVauW7rrrLhmG9Y8fwcHBioqKUlRUlGbOnKl58+Zp8eLFat26tU33/avhw4db/lyzZk2NHTtWPXv21KVLl2x+JgAAcO1i1QLnKnEi261bN40dO1ZTp07Vv//9bwUEBOjWW2+1HN+2bZtq165d4hufOnXKUon9cxL7ZyaTqejAvbzk7e2t3NzcEt+zJP744w/Nnz9frVu3LjKJzcnJsfpgW3Z2tl1jAAAAwNWVuLXgueeek5eXl9q1a6d33nlH77zzjnx8fCzHZ8+erS5dupT4xhkZGTIMQwkJCVb7IyIiFBQUpKCgII0ZM6bQa3Nzc5WSkqKsrCx16NChxPcszpgxYxQYGKjw8HAdOHBAixYtKvLclJQUhYaGWraYmBi7xAAAANwcPbJOVeJENiIiQmvXrtXp06d1+vRp9erVy+r4p59+qvHjx5c5oE2bNiktLU0NGjQosJzXmDFjFBQUpICAAE2dOlVTpkxR9+7dy3xPSRo1apRSU1O1bNkyeXp6ql+/fgVaG65ITk5WVlaWZbO1NxgAAABlZ/MLEUJDQwvdX7lyZZvGiY+Pl8lkUnp6utX+WrVqSZL8/f0LXDNq1CglJiYqKChIkZGRVq0HISEh2r9/f4FrMjMz5enpWWT7whURERGKiIhQ3bp1df311ysmJkYbN27UzTffXOBcX19f+fr6lug5AQDAtYMeWecqcUXW3sLDw9W5c2fNmDFD586dK9E1ERERio+PV1RUVIH+2YSEBO3YsaNAFXfr1q2Ki4uz6UNbZrNZknjBAwAAQDnmskRWkt544w3l5eWpefPm+vjjj7Vz506lp6dr3rx52rVrlzw9PUs8Vt++fWUymdSvXz9t2bJFGRkZmj17tqZNm6Ynn3yyyOt++OEHzZgxQ2lpadq/f79WrlypPn36qHbt2oVWYwEAAIpEj6xT2dxaYE+1a9dWamqqJk+erOTkZB06dEi+vr6qX7++Ro4cqSFDhpR4rLCwMK1bt05jx45Vjx49lJWVpfj4eL366qsaMGBAkdcFBARo4cKFGj9+vM6dO6fo6Gh17dpV48aNo30AAADYxlFJJ4lsoVyayEpSdHS0pk+frunTpxd73r59+646Vt26dbVw4UKb7t+oUSOtXLnSpmsAAADgei5PZAEAACoKPuzlXC7tkQUAAABKi4osAACAvdAj61RUZAEAAOCWqMgCAADYCxVZp6IiCwAAALdERRYAAMBOWLXAuajIAgAAwC1RkQUAALAXemSdioosAAAA3BIVWQAAADuhR9a5SGQBAADshdYCp6K1AAAAAG6JiiwAAIC9UJF1KiqyAAAAcEtUZAEAAOzE9N/NEeOiICqyAAAAcEtUZAEAAOyFHlmnoiILAAAAt0RFFgAAwE54IYJzUZEFAACAW6IiCwAAYC/0yDoVFVkAAAC4JSqyAAAA9kT11GmoyAIAAMAtUZEFAACwE1YtcC4SWQAAAHvhw15ORWsBAAAA3BIVWQAAADuhtcC5qMgCAADALVGRBQAAsBd6ZJ2KiiwAAADcEhVZAAAAO6FH1rlIZO2ovo9JIT4mV4fhlo7lZ7s6BLd11pzj6hDc2o79NVwdglv7v1oHXB2CWzt1+DpXh+CWzpjMGuvqIFAukMgCAADYCz2yTkWPLAAAANwSFVkAAAB7oSLrVFRkAQAA4JaoyAIAANgJqxY4FxVZAAAAuCUqsgAAAPZCj6xTkcgCAADYickwZDLsn3U6YsyKgNYCAAAAuCUqsgAAAPZCa4FTUZEFAACAW6IiCwAAYCcsv+VcVGQBAADglqjIAgAA2As9sk5FRRYAAABuiYosAACAndAj61xUZAEAAOCWqMgCAADYCz2yTkVFFgAAAG6JiiwAAICd0CPrXCSyAAAA9kJrgVPRWgAAAAC3REUWAADAjmgDcB4qsgAAABVISkqKWrRooeDgYFWtWlU9e/ZUenq61Tnt27eXyWSy2h599NES3+PRRx+VyWTStGnT7By9bUhkAQAA7MUwHLeV0Jo1a5SUlKSNGzfq22+/1aVLl9SlSxedO3fO6ryBAwfq6NGjlu3FF18s0fhffPGFNm7cqGrVqtk0NY5AawEAAEAFsnTpUquv586dq6pVq2rLli1q27atZX9AQICioqJsGvvw4cN67LHH9M0336h79+52ibcsqMgCAADYyZXltxyxSVJ2drbVlpOTc9WYsrKyJEmVK1e22j9//nxFRESoYcOGSk5O1vnz54sdx2w266GHHtKoUaPUoEGD0k2QnVGRBQAAcBMxMTFWX48fP14TJkwo8nyz2awnnnhCbdq0UcOGDS37H3jgAdWsWVPVqlXTtm3bNGbMGKWnp2vhwoVFjjV16lR5eXnp8ccfL/Nz2AuJLAAAgL04eB3ZgwcPKiQkxLLb19e32MuSkpK0fft2fffdd1b7Bw0aZPlzo0aNFB0drY4dO2r37t2qXbt2gXG2bNmi1157TVu3bpXJZCrDg9gXrQUAAABuIiQkxGorLpEdOnSolixZolWrVql69erFjtuqVStJUkZGRqHH161bpxMnTqhGjRry8vKSl5eX9u/fryeffFKxsbGlfp6yoiILAABgJybz5c0R45aUYRh67LHH9MUXX2j16tWKi4u76jVpaWmSpOjo6EKPP/TQQ+rUqZPVvttvv10PPfSQ+vfvX/Lg7IxEFgAAoAJJSkrSggULtGjRIgUHB+vYsWOSpNDQUPn7+2v37t1asGCBunXrpvDwcG3btk3Dhw9X27Zt1bhxY8s49erVU0pKinr16qXw8HCFh4db3cfb21tRUVFKSEhw6vP9GYksAACAvTi4R7YkZs2aJenySw/+bM6cOUpMTJSPj4+WL1+uadOm6dy5c4qJidE999yjcePGWZ2fnp5uWfGgvCKRBQAAqECMq7w8ISYmRmvWrCnzOPv27bMlLIcgkQUAALCTP6/5au9xUZDLVy04duyYhg0bpvj4ePn5+SkyMlJt2rTRrFmzrBbmjY2NtbwLODAwUE2bNtWnn35qOZ6YmKiePXsWGH/16tUymUzKzMy8aiw5OTm68cYbZTKZLE3PAAAAJVYOXlF7LXFpIrtnzx41adJEy5Yt0+TJk5WamqoNGzZo9OjRWrJkiZYvX251/qRJk3T06FGlpqaqRYsW6t27t9avX2+3eEaPHl0u3hsMAACAq3Npa8GQIUPk5eWlzZs3KzAw0LK/Vq1auuuuuwr0ZgQHBysqKkpRUVGaOXOm5s2bp8WLF6t169ZljuXrr7/WsmXL9Pnnn+vrr78u83gAAODaQ2uBc7kskT116pSlEvvnJPbPintzhJeXl7y9vZWbm1vmWI4fP66BAwfq3//+twICAq56fk5OjtW7jbOzs8scAwAAAGzjstaCjIwMGYZRYO2xiIgIBQUFKSgoSGPGjCn02tzcXKWkpCgrK0sdOnQoUxyGYSgxMVGPPvqomjdvXqJrUlJSFBoaatn++t5jAABwjTIcuKEAl3/Y6682bdqktLQ0NWjQwKrqKUljxoxRUFCQAgICNHXqVE2ZMkXdu3cv0/2mT5+uM2fOKDk5ucTXJCcnKysry7IdPHiwTDEAAADAdi5rLYiPj5fJZFJ6errV/lq1akmS/P39C1wzatQoJSYmKigoSJGRkVatByEhIdq/f3+BazIzM+Xp6Vlk+8LKlSu1YcOGAu8qbt68ufr27av33nuvwDW+vr7FvtsYAABcm+iRdS6XVWTDw8PVuXNnzZgxQ+fOnSvRNREREYqPj1dUVFSB/tmEhATt2LGjQBV369atiouLk7e3d6Fjvv766/rpp5+UlpamtLQ0/ec//5Ekffzxx3rhhRdK8WQAAABwBpe2FrzxxhvKy8tT8+bN9fHHH2vnzp1KT0/XvHnztGvXLnl6epZ4rL59+8pkMqlfv37asmWLMjIyNHv2bE2bNk1PPvlkkdfVqFFDDRs2tGx169aVJNWuXVvVq1cv8zMCAIBrCOvIOpVLl9+qXbu2UlNTNXnyZCUnJ+vQoUPy9fVV/fr1NXLkSA0ZMqTEY4WFhWndunUaO3asevTooaysLMXHx+vVV1/VgAEDHPgUAAAAcAWXv6I2Ojpa06dP1/Tp04s9ryTv861bt64WLlxYpnhiY2Ov+m5hAACAwtAj61zlbtUCAAAAoCRcXpEFAACoMBy15isV2UJRkQUAAIBboiILAABgJ/TIOheJLAAAgL2YjcubI8ZFAbQWAAAAwC1RkQUAALAXPuzlVFRkAQAA4JaoyAIAANiJSQ76sJf9h6wQqMgCAADALVGRBQAAsBfDuLw5YlwUQEUWAAAAbomKLAAAgJ3wQgTnoiILAAAAt0RFFgAAwF5YR9apqMgCAADALVGRBQAAsBOTYcjkgBUGHDFmRUAiCwAAYC/m/26OGBcF0FoAAAAAt0RFFgAAwE5oLXAuKrIAAABwS1RkAQAA7IXlt5yKiiwAAADcEhVZAAAAezGMy5sjxkUBVGQBAADglqjIAgAA2InJuLw5YlwUREUWAAAAbomKLAAAgL3QI+tUVGQBAADglqjIAgAA2InJfHlzxLgoiIosAAAA3BIVWQAAAHuhR9apSGQBAADshVfUOhWJrB1tuuijQG9PV4fhlqI8s10dgttan1PZ1SG4taY+v7s6BLc26qf7XB2CW3vphsOuDsEteWdnSwp1dRgoB0hkAQAA7MRkGDI5oA3AEWNWBHzYCwAAAG6JiiwAAIC98GEvp6IiCwAAALdERRYAAMBeDEmOeHkBBdlCUZEFAACAW6IiCwAAYCesWuBcVGQBAADglqjIAgAA2IshB61aYP8hKwIqsgAAAHBLVGQBAADshXVknYpEFgAAwF7MkkwOGhcF0FoAAAAAt0RFFgAAwE5Yfsu5qMgCAADALVGRBQAAsBc+7OVUVGQBAADglqjIAgAA2AsVWaeiIgsAAAC3REUWAADAXqjIOhUVWQAAALglKrIAAAD2wpu9nIqKLAAAANwSFVkAAAA74c1ezkVFFgAAAG6JiiwAAIC9sGqBU5HIAgAA2IvZkEwOSDrNJLKFobUAAAAAbomKLAAAgL3QWuBUVGQBAADglqjIAgAA2I2DKrKiIlsYKrIAAABwS1RkAQAA7IUeWaeiIgsAAAC35PJE9tixYxo2bJji4+Pl5+enyMhItWnTRrNmzdL58+ct58XGxspkMslkMikwMFBNmzbVp59+ajmemJionj17Fhh/9erVMplMyszMLDKGP499ZZsyZYo9HxMAAFwLzIbjthJKSUlRixYtFBwcrKpVq6pnz55KT0+3Oqd9+/YFcp9HH320yDEvXbqkMWPGqFGjRgoMDFS1atXUr18/HTlypNRTZQ8uTWT37NmjJk2aaNmyZZo8ebJSU1O1YcMGjR49WkuWLNHy5cutzp80aZKOHj2q1NRUtWjRQr1799b69evtEsuVsa9sjz32mF3GBQAAcKY1a9YoKSlJGzdu1LfffqtLly6pS5cuOnfunNV5AwcOtMp9XnzxxSLHPH/+vLZu3apnnnlGW7du1cKFC5Wenq4ePXo4+nGK5dIe2SFDhsjLy0ubN29WYGCgZX+tWrV01113yfhLP0hwcLCioqIUFRWlmTNnat68eVq8eLFat25d5liujA0AAFBqhvny5ohxS2jp0qVWX8+dO1dVq1bVli1b1LZtW8v+gICAEuc+oaGh+vbbb632zZgxQy1bttSBAwdUo0aNEsdnTy6ryJ46dUrLli1TUlKSVRL7ZyaTqcjrvby85O3trdzcXLvEM2XKFIWHh6tJkyZ66aWXlJeXZ5dxAQAA7CU7O9tqy8nJueo1WVlZkqTKlStb7Z8/f74iIiLUsGFDJScnW7V0lkRWVpZMJpPCwsJsus6eXFaRzcjIkGEYSkhIsNofERGhixcvSpKSkpI0derUAtfm5ubqlVdeUVZWljp06FDmWB5//HE1bdpUlStX1vr165WcnKyjR4/q1VdfLfT8nJwcq2+c7OzsMscAAAAqAAevWhATE2O1e/z48ZowYUKRl5nNZj3xxBNq06aNGjZsaNn/wAMPqGbNmqpWrZq2bdumMWPGKD09XQsXLixROBcvXtSYMWPUp08fhYSE2P48dlLult/atGmTzGaz+vbtW+CnjDFjxmjcuHG6ePGigoKCNGXKFHXv3r3M9xwxYoTlz40bN5aPj48GDx6slJQU+fr6Fjg/JSVFEydOLPN9AQAAbHHw4EGrxLGwPOXPkpKStH37dn333XdW+wcNGmT5c6NGjRQdHa2OHTtq9+7dql27drFjXrp0Sffff78Mw9CsWbNK8RT247JENj4+XiaTqcCn6GrVqiVJ8vf3L3DNqFGjlJiYqKCgIEVGRlq1HoSEhGj//v0FrsnMzJSnp2eR7QuFadWqlfLy8rRv374CFWNJSk5Otkp+s7OzC/yEBAAArkFmQw55C9d/Vy0ICQkpcQV06NChWrJkidauXavq1asXe26rVq0kXf6NeXGJ7JUkdv/+/Vq5cqVLq7GSC3tkw8PD1blzZ82YMaPAp+iKEhERofj4eEVFRRXon01ISNCOHTsKVHG3bt2quLg4eXt7lzi2tLQ0eXh4qGrVqoUe9/X1tXwj2fINBQAAKrgrrQWO2EocgqGhQ4fqiy++0MqVKxUXF3fVa9LS0iRJ0dHRRZ5zJYn97bfftHz5coWHh5c4Jkdx6fJbb7zxhvLy8tS8eXN9/PHH2rlzp9LT0zVv3jzt2rVLnp6eJR6rb9++MplM6tevn7Zs2aKMjAzNnj1b06ZN05NPPlnkdRs2bNC0adP0008/ac+ePZo/f76GDx+uBx98UJUqVbLHYwIAADhNUlKS5s2bpwULFig4OFjHjh3TsWPHdOHCBUnS7t279dxzz2nLli3at2+fvvzyS/Xr109t27ZV48aNLePUq1dPX3zxhaTLSey9996rzZs3a/78+crPz7eMa68P3peGS3tka9eurdTUVE2ePFnJyck6dOiQfH19Vb9+fY0cOVJDhgwp8VhhYWFat26dxo4dqx49eigrK0vx8fF69dVXNWDAgCKv8/X11UcffaQJEyYoJydHcXFxGj58uFXrAAAAQIkYctCHvUp+6pW+1fbt21vtnzNnjhITE+Xj46Ply5dr2rRpOnfunGJiYnTPPfdo3LhxVuenp6dbVjw4fPiwvvzyS0nSjTfeaHXeqlWrCtzLWUzGXxdrhc2ys7MVGhqqL3+qrcDgkleR8T/BHhddHYLbOpZPa0tZNPX53dUhuLVpp25xdQhu7aUbPr36SSjgyv93s7Kyyk1735WYOkUPlpeHj93HzzPnavnRt8rVM5cH5W7VAgAAALfl4OW3YM2lPbIAAABAaVGRBQAAsBezWZIDXlFrdsCYFQAVWQAAALglKrIAAAD2Qo+sU1GRBQAAgFuiIgsAAGAvVGSdioosAAAA3BIVWQAAAHsxG7LpNVw2jYu/IpEFAACwE8MwyzDsv1SWI8asCGgtAAAAgFuiIgsAAGAvhuGYNgA+7FUoKrIAAABwS1RkAQAA7MVw0Ie9qMgWioosAAAA3BIVWQAAAHsxmyWTA1YYYNWCQlGRBQAAgFuiIgsAAGAv9Mg6FRVZAAAAuCUqsgAAAHZimM0yHNAjy5u9CkdFFgAAAG6JiiwAAIC90CPrVFRkAQAA4JaoyAIAANiL2ZBMVGSdhUQWAADAXgxDkiNeiEAiWxhaCwAAAOCWqMgCAADYiWE2ZDigtcCgIlsoKrIAAABwS1RkAQAA7MUwyzE9srwQoTBUZAEAAOCWqMgCAADYCT2yzkVFFgAAAG6JiiwAAIC90CPrVCSydnCl3H/+LN9kpWXyYO5K63x+vqtDcGtnfPjeK4ucs5dcHYJby87OdnUIbunKvJXHX7fn6ZLkgLDyxN+1wpiM8vhd4GYOHTqkmJgYV4cBAMA15eDBg6pevbqrw5AkXbx4UXFxcTp27JjD7hEVFaW9e/fKz8/PYfdwNySydmA2m3XkyBEFBwfLZDK5OpwCsrOzFRMTo4MHDyokJMTV4bgV5q5smL+yYf7KhvkrvfI+d4Zh6MyZM6pWrZo8PMrPx30uXryo3Nxch43v4+NDEvsXtBbYgYeHR7n5ibA4ISEh5fIfJHfA3JUN81c2zF/ZMH+lV57nLjQ01NUhFODn50ei6WTl58cYAAAAwAYksgAAAHBLJLLXAF9fX40fP16+vr6uDsXtMHdlw/yVDfNXNsxf6TF3cBd82AsAAABuiYosAAAA3BKJLAAAANwSiSwAAADcEoksAAAA3BKJLK5J+fn5rg6hwuDzomXD/Nnm7NmzzFkpHT58WL/88ourwwDsikTWDWVkZOj555/X+PHj9a9//cvV4biVkydPSpI8PT1JZkvhwIEDmj17tl5//XUtXrxYksrla5nLq927d+vpp5/W4MGD9fLLL0ti/myxa9cu3XDDDVqwYIGrQ3E7qampatiwoTIyMlwdCmBXvKLWzezYsUNt2rRRs2bNdOHCBf3000/68MMPNXHiRN18883l6p3T5c3OnTvVpEkT3X333VqwYIElmfX09HR1aG7h559/Vvfu3VW7dm0dOXJE58+fV0ZGhoYPH+7q0NzCtm3b1KVLF7Vs2VJeXl6aO3euDMPQqFGjXB2a2/joo4+0d+9eJSUlyWw266GHHnJ1SG7hp59+Utu2bfXII4+oR48eBY4bhsEPVHBbZD1u5OLFixozZoz69OmjFStWaPXq1dq+fbuOHj2qUaNGacWKFfzKrQhHjhzR3//+dzVu3FjfffedHn74YUlUZktq79696tGjh/r06aNly5Zp+fLlGjRokJYsWaLff/+d77ur+O2333TXXXepf//++vLLL7VgwQL169dPOTk5rg7NrTRp0kRDhw7VM888o/79++v999+3HOPvceF++eUX3XrrrXr00Uf1z3/+U/n5+VqzZo2+/PJLrVixQhK/FYB7oyLrRvz8/HT27FnFxMRIkjw8PBQXF6e1a9eqa9euGj9+vOrUqaPY2FjXBloOLV++XNWqVdPjjz+uQ4cOafTo0Xr44Yf13nvvydPTU3l5efLy4q9DYfLy8vTBBx+oUaNGevbZZ+Xt7a2YmBi1bt1ar7zyis6cOaOIiAhXh1lumc1mzZw5UzfddJMmTJgg6fLf5ZycHK1Zs0abN29WzZo1NWLECNWsWdO1wZZzkZGRWrFihbZs2aIjR47okUceUVhYmNasWaMaNWpo2LBhrg6xXDGbzRo1apTMZrP69eun/Px89ejRQ8ePH9e+fft0/vx59e7dW2+99ZZ8fHxcHS5QKvyf243k5uYqNzfX0uPk5eWl3NxchYeH65tvvlH9+vX1wgsv6J133nFxpOVPjx49FBwcrHbt2ik3N1eGYWjMmDGWZNbLy4s2gyJ4enqqXr16CgsLU2BgoKTLv4ps0qSJQkJCdO7cuQLX8KvK//Hw8NAzzzyjbdu2WV73OWXKFM2bN09Dhw5VdHS0XnvtNf3666/6+uuvXRxt+WUYhmJjYxUQEKDz58/rlVdeUVhYmHr16qXAwED98MMPrg6x3PHw8NBbb72l++67T48//rhOnz6tqKgovfvuuwoMDNSePXvUu3dvBQcH6/XXX3d1uEDpGHALZrPZMAzDWLJkieHn52e8/fbblmMXLlwwDMMwPv74Y6NGjRrGnj17LOfDKHQuzp07Z8ybN8+oVq2a0a9fP8v+OXPmGHv37nVidO7h9OnTlj9fmc+cnByjVq1axubNmy3H1qxZ4+zQyr38/Hyrr3fv3m3cd999xtKlSy37fvrpJ8NkMjF/JdCuXTtj48aNhmEYxkMPPWSEhIQYXl5exscff+ziyMqfvLw8wzAM4+DBg0aTJk2Mm2++2di/f7/VOTNmzDBq1Khh7Nu3j/9vwC1RkXUTV6pbbdq0UVJSklJSUuTt7a3ExET5+flJkgICAuTr66vAwECqYX9S2FwEBASoZ8+ekmRpMwgPD9e0adO0d+9eJ0dY/oWFhVn+bDKZZDablZ2drXPnzllaMsaNG6fJkyfryJEjioyM5Hvwv/76AcxatWpp1qxZCg8Pt+z7448/1LBhQ9WoUcPZ4bmNK78xCQsL04EDB7RgwQItX75c3333nT777DP97W9/kyTdf//9Lo60/LjyGYDq1avrm2++0erVqxUdHV3gvJCQEIWHh/N3Fm6JRLYcMwr59WxYWJgeeeQRXbx4UcnJyTpx4oSGDh2qvLw8bdy4UYGBgfR6qmS/2g4MDFSvXr2Un5+vxMREVapUST/++CN9iirZ/F05JygoSCkpKXrttde0adMmRUVFOSnK8quo+buyv3Llylb7ly1bpvDwcIWEhDgrxHKtsPm78gNBu3bt9OCDD6pq1apasmSJGjVqpEaNGsnLy0uNGjVyRbjlyl/nztPTU2azWVWqVNF9991X4Pxff/1V9erVY8UbuC2TYfBx4/Lm119/1fHjx3XrrbcW+T/EvXv36vPPP9f48eMVGRmpoKAgHT9+XF9//bWaNm3qgqjLh5LM3V8NGDBAn3zyiX744QfVr1/fCVGWX7bM36VLl9SyZUuFh4dr3bp1+v7779W8eXMnRlv+2Pr9t2fPHv3rX//SzJkztXbtWjVu3NhJkZZPJZm/VatW6d1339Xo0aN1ww03uCDK8snW773du3drzpw5mjlzptatW6eGDRs6KVLAzpzfzYDi5OfnG8OGDTNMJpOxatUqwzAK7/G8Yvfu3cZ7771nfP7559d8b6etc2cYhrFo0aICfZ7XKlvn7+jRo4aXl5fh4+Nj/PTTT06Ksvyydf62bdtmPPLII0adOnWM1NRU5wRZjtkyf2fOnHFiZOWfrd97P/30k/HQQw8ZNWvW5HsPbo9Ethw6efKkMXjwYMPPz89YsWKFYRiF/6P01w+RoORzd0VWVpZx+PBhZ4VX7tkyfzk5OcbLL79s7Nq1y5khlmu2zF9mZqbx/fffGwcOHHBmiOXa1eaPDyMVzZbvvdOnTxurVq0y9u3b58wQAYcgkS1HrnzC1DAM49SpU8Yjjzxi9Y8SiWvRSjN3zOf/lPZ7Lzc31ynxlXf83S0b5q/0mDtc60hky4GsrCzLn4v6R2ndunWGYfCP0l8xd2XD/JUN81c2zF/pMXfAZSSyLnb06FGja9euxuuvv27Z99d/lB5++GEjNDTU+OWXX1wRYrnF3JUN81c2zF/ZMH+lx9wB/8N6Gy6WnZ0tf39/ffLJJ3r33Xcl/W/tP0mqXLmynn32WTVt2lQLFiyQcfmHD1eGXG4wd2XD/JUN81c2zF/pMXfA/5DIuojZbJZhGKpbt65eeeUV1a5dW//617+s/lHKy8uTdHkB9YCAAO3fv18mk+maX7SauSsb5q9smL+yYf5Kj7kDCiKRdYGMjAw99dRTuvfeezVjxgwFBgYqJSVF9erV0+zZs/XOO+9Ikry8vHTp0iVJUnh4uOrUqXPN/2TN3JUN81c2zF/ZMH+lx9wBRXBG/wL+Jy0tzahSpYpx2223Gc2aNTM8PT2N++67z8jOzjZ+++034+GHHzZuuukm48UXXzQMwzBOnDhhTJgwwahataqRnp7u4uhdi7krG+avbJi/smH+So+5A4pGIutE27dvNwIDA43nnnvO0pj/5ptvGiaTyfjss88MwzCM9PR048knnzSqVatmXHfddcYtt9xi1KtX75pftJq5Kxvmr2yYv7Jh/kqPuQOKxytqnSQzM1PXX3+9YmJitGLFCgUHB0uS8vLydP3112vw4MEaOXKkJOn06dM6evSovvrqKyUkJOjGG29UjRo1XBm+SzF3ZcP8lQ3zVzbMX+kxd0AJuDqTvpaMGzfOqFWrlvHCCy9Y3uazY8cOw8vLy1i0aJGLoyvfmLuyYf7KhvkrG+av9Jg7oHherk6kryXPPfecvL29NWvWLAUHB6tVq1a655579I9//EM9evRwdXjlGnNXNsxf2TB/ZcP8lR5zBxSP1gIHOnTokNasWaPz58+rc+fOio2NlSRNmDBB77zzjs6ePatevXpp7ty5kqT8/Hx5enq6LuByhLkrG+avbJi/smH+So+5A2zD8lsOsmPHDt15551aunSpMjIyLP8YSZf/QRo2bJhMJpPq1Kmj48ePSxL/GP0Xc1c2zF/ZMH9lw/yVHnMHlIKrexsqou3btxuVKlUyxo0bZ/U+7C+//NJYuHCh5evx48cbMTExRkpKinHo0CFXhFruMHdlw/yVDfNXNsxf6TF3QOmQyNrZqVOnjLZt2xpDhw612j9lyhTDZDIZHTp0sPpHaeLEiUZAQIDxyiuvWL0r+1rE3JUN81c2zF/ZMH+lx9wBpUcia2e//PKLUbt2bWPlypVGfn6+YRiGMWvWLMPb29uYOXOm0blzZ6Nbt25W/yhNnTrV+PXXX10VcrnB3JUN81c2zF/ZMH+lx9wBpUcia2cffPCB4enpaZjNZsu+gwcPGmvXrjUMwzB+/vlno2PHjkbLli1ZrPovmLuyYf7KhvkrG+av9Jg7oPT4sJedxcbGysvLS1988YUkyTAMVa9eXbfeeqvMZrMaNmyo3r17yzAMRUVFuTja8oW5Kxvmr2yYv7Jh/kqPuQNKj0TWzmJjYxUaGqr33ntP+/fvl8lkshzz8Lg83enp6YqNjVVgYKCrwiyXmLuyYf7KhvkrG+av9Jg7oAxcUwiu2D777DPDx8fHeOihh4wdO3ZY9mdlZRmjRo0yKlWqZGzfvt2FEZZfzF3ZMH9lw/yVDfNXeswdUDq8EMEB8vPz9e6772ro0KGKj49X69at5e3trcOHD2vz5s36z3/+oyZNmrg6zHKJuSsb5q9smL+yYf5Kj7kDSodE1oF++OEHvfjii9q9e7eCg4N1yy23aMCAAYqPj3d1aOUec1c2zF/ZMH9lw/yVHnMH2IZE1sF4fWDpMXdlw/yVDfNXNsxf6TF3QMnxYS8Hu9KoL13+JCpKjrkrG+avbJi/smH+So+5A0qOiiwAAADcEhVZAAAAuCUSWQAAALglElkAAAC4JRJZAAAAuCUSWQAAALglElkAAAC4JRJZAAAAuCUSWQDlWmxsrKZNm+bw++zbt08mk0lpaWkOvxcAwD5IZAEUKzExUSaTSSaTSd7e3oqMjFTnzp01e/Zsmc1mu91n7ty5CgsLK7D/xx9/1KBBg+x2H+nyM/Xs2dNqX0xMjI4ePaqGDRva9V6Fyc7O1jPPPKMGDRrI399f4eHhatGihV588UWdPn3acl779u0tc+/n56f69evrjTfesByfMGGCbrzxxgLjk5QDuFaQyAK4qq5du+ro0aPat2+fvv76a912220aNmyY7rzzTuXl5Tn03lWqVFFAQIBD7yFJnp6eioqKkpeXl0Pv88cff+imm27SnDlzNHLkSP3www/aunWrXnjhBaWmpmrBggVW5w8cOFBHjx7VL7/8ovvvv19JSUn68MMPHRojALgLElkAV+Xr66uoqChdd911atq0qZ566iktWrRIX3/9tebOnWs5LzMzU4888oiqVKmikJAQdejQQT/99JPl+E8//aTbbrtNwcHBCgkJUbNmzbR582atXr1a/fv3V1ZWlqUCOWHCBEkFWwtMJpPeffdd9erVSwEBAapTp46+/PJLy/H8/HwNGDBAcXFx8vf3V0JCgl577TXL8QkTJui9997TokWLLPdavXp1oVXMNWvWqGXLlvL19VV0dLTGjh1rlbi3b99ejz/+uEaPHq3KlSsrKirKEndRnnrqKR04cECbNm1S//791bhxY9WsWVNdunTRhx9+qCFDhlidHxAQoKioKNWqVUsTJkwo8LwAcC0jkQVQKh06dNANN9yghQsXWvbdd999OnHihL7++mtt2bJFTZs2VceOHfXHH39Ikvr27avq1avrxx9/1JYtWzR27Fh5e3urdevWmjZtmkJCQnT06FEdPXpUI0eOLPLeEydO1P33369t27apW7du6tu3r+UeZrNZ1atX16effqpffvlFzz77rJ566il98sknkqSRI0fq/vvvt1SZjx49qtatWxe4x+HDh9WtWze1aNFCP/30k2bNmqV//etfev75563Oe++99xQYGKgffvhBL774oiZNmqRvv/220LjNZrM+/vhjPfjgg6pWrVqh55hMpmJmXfL391dubm6x5wDAtYJEFkCp1atXT/v27ZMkfffdd9q0aZM+/fRTNW/eXHXq1NHLL7+ssLAwffbZZ5KkAwcOqFOnTqpXr57q1Kmj++67TzfccIN8fHwUGhoqk8mkqKgoRUVFKSgoqMj7JiYmqk+fPoqPj9fkyZN19uxZbdq0SZLk7e2tiRMnqnnz5oqLi1Pfvn3Vv39/SyIbFBQkf39/S5U5KipKPj4+Be7xxhtvKCYmRjNmzFC9evXUs2dPTZw4Ua+88opVb3Djxo01fvx41alTR/369VPz5s21YsWKQuM+efKkMjMzlZCQYLW/WbNmCgoKUlBQkPr06VPotfn5+Zo3b562bdumDh06FDk3AHAtcWwzGIAKzTAMSwXxp59+0tmzZxUeHm51zoULF7R7925J0ogRI/TII4/ogw8+UKdOnXTfffepdu3aNt+3cePGlj8HBgYqJCREJ06csOybOXOmZs+erQMHDujChQvKzc0t9ENRxdm5c6duvvlmqwppmzZtdPbsWR06dEg1atQoEIskRUdHW8VSEl988YVyc3M1ZswYXbhwwerYG2+8oXfffVe5ubny9PTU8OHD9Y9//MOm8QGgoiKRBVBqO3fuVFxcnCTp7Nmzio6O1urVqwucd2U1ggkTJuiBBx7QV199pa+//lrjx4/XRx99pF69etl0X29vb6uvTSaTpUr60UcfaeTIkXrllVd08803Kzg4WC+99JJ++OEH2x+wjLH8VZUqVRQWFqb09HSr/VeS4uDgYGVmZlod69u3r55++mn5+/srOjpaHh7/+0VaSEiIsrKyCtznyhihoaG2Pg4AuBVaCwCUysqVK/Xzzz/rnnvukSQ1bdpUx44dk5eXl+Lj4622iIgIy3V169bV8OHDtWzZMt19992aM2eOJMnHx0f5+flljuv7779X69atNWTIEDVp0kTx8fGWivAVJbnX9ddfrw0bNsgwDKuxg4ODVb169VLF5uHhofvvv1/z5s3TkSNHSnRNaGio4uPjdd1111klsZKUkJCgQ4cO6fjx41b7t27dKj8/P0uCDAAVFYksgKvKycnRsWPHdPjwYW3dulWTJ0/WXXfdpTvvvFP9+vWTJHXq1Ek333yzevbsqWXLlmnfvn1av369nn76aW3evFkXLlzQ0KFDtXr1au3fv1/ff/+9fvzxR11//fWSLq9OcPbsWa1YsUK///67zp8/X6pY69Spo82bN+ubb77Rr7/+qmeeeUY//vij1TmxsbHatm2b0tPT9fvvv+vSpUsFxhkyZIgOHjyoxx57TLt27dKiRYs0fvx4jRgxokBCaYvJkyfruuuuU8uWLTV79mxt27ZNu3fv1hdffKENGzbI09OzxGPdfvvtSkhIUJ8+fbR+/Xrt2bNHn332mcaNG6dhw4bZNBYAuCNaCwBc1dKlSxUdHS0vLy9VqlRJN9xwg15//XU9/PDDlqTOZDLpP//5j55++mn1799fJ0+eVFRUlNq2bavIyEh5enrq1KlT6tevn44fP66IiAjdfffdmjhxoiSpdevWevTRR9W7d2+dOnVK48ePv+pSVoUZPHiwUlNT1bt3b5lMJvXp00dDhgzR119/bTln4MCBWr16tZo3b66zZ89q1apVio2NtRrnuuuu03/+8x+NGjVKN9xwgypXrqwBAwZo3LhxpZ5HSQoPD9emTZs0depUvfTSS9q7d688PDxUp04d9e7dW0888USJx/Ly8tKyZcv01FNPqU+fPjp58qTi4uI0bNgwjRgxokxxAoA7MBl//r0ZAAAA4CZoLQAAAIBbIpEFAACAWyKRBQAAgFsikQUAAIBbIpEFAACAWyKRBQAAgFsikQUAAIBbIpEFAACAWyKRBQAAgFsikQUAAIBbIpEFAACAWyKRBQAAgFv6f5MDVmW+87ZJAAAAAElFTkSuQmCC", | |
| "text/plain": [ | |
| "<Figure size 800x600 with 2 Axes>" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "import matplotlib.pyplot as plt\n", | |
| "import numpy as np\n", | |
| "\n", | |
| "# Create a copy of speed_matrix and set diagonal to None to ignore same-GPU transfers\n", | |
| "masked_matrix = speed_matrix.copy()\n", | |
| "for i in range(len(list_of_gpus)):\n", | |
| " masked_matrix[i,i] = np.nan\n", | |
| "\n", | |
| "fig, ax = plt.subplots(figsize=(8, 6))\n", | |
| "im = ax.imshow(masked_matrix, cmap='viridis')\n", | |
| "\n", | |
| "# Add labels\n", | |
| "ax.set_xticks(np.arange(len(list_of_gpus)))\n", | |
| "ax.set_yticks(np.arange(len(list_of_gpus)))\n", | |
| "ax.set_xticklabels([f'GPU {i}' for i in range(len(list_of_gpus))])\n", | |
| "ax.set_yticklabels([f'GPU {i}' for i in range(len(list_of_gpus))])\n", | |
| "\n", | |
| "# Add colorbar\n", | |
| "cbar = plt.colorbar(im)\n", | |
| "cbar.set_label('GB/s')\n", | |
| "\n", | |
| "# Add title and axis labels\n", | |
| "plt.title('GPU to GPU Transfer Speeds')\n", | |
| "plt.xlabel('Destination GPU')\n", | |
| "plt.ylabel('Source GPU')\n", | |
| "\n", | |
| "# Rotate x-axis labels for better readability\n", | |
| "plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\")\n", | |
| "\n", | |
| "plt.tight_layout()\n", | |
| "plt.show()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3 (ipykernel)", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.10.15" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 2 | |
| } |
driver565.57.1 enabled resize-bar， question：CUDA error at simpleP2P.cu:129 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"。 why this
You might have to enable resizable bar in your BIOS. it should be under pci-e settings
hi，
The Resizeable Bar has been started in the BIOS, and the ACS for PCI-E has been disabled and the IOMMU has been turned off.
Steps:
1、./NVIDIA-Linux-x86_64-550.90.07.run --no-kernel-modules
2、download 550.90.07-p2p
3、./install.sh
4. deviceQuery is normal
5. simpleP2P will report the error mentioned above
Can you help me know what to look out for?
| |
yingzilnn01
|
|
***@***.***
|
---- Replied Message ----
| From | Ștefan-Gabriel ***@***.***> |
| Date | 12/20/2024 19:43 |
| To | ***@***.***> |
| Cc | ***@***.***> |
| Subject | Re: legraphista/readme.md |
@legraphista commented on this gist.
driver565.57.1 enabled resize-bar， question：CUDA error at simpleP2P.cu:129 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"。 why this
You might have to enable resizable bar in your BIOS. it should be under pci-e settings
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS or Android.
I've never had this issue, try posting in https://github.com/tinygrad/open-gpu-kernel-modules and ask for help there. They most likely know better
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
driver565.57.1 enabled resize-bar， question：CUDA error at simpleP2P.cu:129 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)"。 why this