Skip to content

Instantly share code, notes, and snippets.

@zhanghang1989

zhanghang1989/download_imagenet.sh

Last active May 7, 2020 04:18
Show Gist options
  • Save zhanghang1989/e97124f43577cbe1a897fa69517e52a7 to your computer and use it in GitHub Desktop.
Save zhanghang1989/e97124f43577cbe1a897fa69517e52a7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
download_imagenet.sh
aws s3 cp s3://zhiz-data/datasets/imagenet/ILSVRC2012_img_train.tar .
aws s3 cp s3://zhiz-data/datasets/imagenet/ILSVRC2012_img_val.tar .
cd ResNeSt/
python scripts/dataset/prepare_imagenet.py --download-dir ~/
Raw
restart.sh
cd ~/
sudo mkdir -p /media/ramdisk
sudo mount -t tmpfs -o size=200G tmpfs /media/ramdisk
cp -r /home/ubuntu/data/ILSVRC2012/ /media/ramdisk
Raw
setup_ec2_horovod.sh
mkdir ~/data
mkdir ~/data/ILSVRC2012
cd ~/data/ILSVRC2012
aws s3 sync s3://zhiz-data/datasets/imagenets/ .
cd ~/
sudo mkdir -p /media/ramdisk
sudo mount -t tmpfs -o size=200G tmpfs /media/ramdisk
cp -r /home/ubuntu/data/ILSVRC2012/ /media/ramdisk
pip install mxnet-cu100
HOROVOD_GPU_ALLREDUCE=NCCL pip install -v --no-cache-dir horovod
pip install --no-cache mpi4py
Raw
setup_pth.sh
aws s3 cp s3://zhiz-data/datasets/imagenet/ILSVRC2012_img_train.tar .
aws s3 cp s3://zhiz-data/datasets/imagenet/ILSVRC2012_img_val.tar .
conda install -y pytorch torchvision cudatoolkit=10.0 -c pytorch
cd ~/
mkdir -p git
cd git/
git clone https://github.com/zhanghang1989/ResNeSt
cd ResNeSt/ && python setup.py develop
python scripts/dataset/prepare_imagenet.py --download-dir ~/
@zhanghang1989
Copy link
Author

zhanghang1989 commented Apr 13, 2020 •
edited
Loading

wget -qO- https://gist.githubusercontent.com/zhanghang1989/e97124f43577cbe1a897fa69517e52a7/raw/9d004f34f62702d960641de5c6e6d987fcdf2903/setup_pth.sh | bash

@zhanghang1989
Copy link
Author

zhanghang1989 commented Apr 22, 2020 •
edited
Loading

wget -qO- https://gist.githubusercontent.com/zhanghang1989/e97124f43577cbe1a897fa69517e52a7/raw/44ba19396994a6ed44116982ef5145cbb87c2714/setup_ec2_horovod.sh | bash

@zhanghang1989
Copy link
Author

wget -qO- https://gist.githubusercontent.com/zhanghang1989/e97124f43577cbe1a897fa69517e52a7/raw/176009c20c1c2721355839ea85ec1eb5c5fb43dc/install_ag.sh | bash

@zhanghang1989
Copy link
Author

wget -qO- https://gist.githubusercontent.com/zhanghang1989/e97124f43577cbe1a897fa69517e52a7/raw/8b60aec7dad1fbf8ffde7301af2d94245b7c651f/setup_pth.sh | bash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment