Skip to content

Instantly share code, notes, and snippets.

View iakashpaul's full-sized avatar
🛠️

Akash iakashpaul

🛠️
36 followers · 90 following
View GitHub Profile
@iakashpaul
iakashpaul / sglang.md
Created September 11, 2024 18:40
sglang_launch_server

Run SGLang with model weights being stored on persistent storage on /data at FP8 Quantization level, roughly 128tok/s for BS_1

HF_HOME=/data python3 -m sglang.launch_server --model NousResearch/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --random-seed 1337 --dtype bfloat16 --quantization fp8
@iakashpaul
iakashpaul / benchmark.md
Last active February 22, 2024 08:37
chsasank-benchmarking

Runs for dtypes

DEVICE=cuda && DTYPE=float32 && python benchmark.py --device ${DEVICE}  --dtype ${DTYPE} > H100_${DEVICE}_${DTYPE}.log
DEVICE=cuda && DTYPE=float16 && python benchmark.py --device ${DEVICE}  --dtype ${DTYPE} > H100_${DEVICE}_${DTYPE}.log
DEVICE=cuda && DTYPE=bfloat16 && python benchmark.py --device ${DEVICE}  --dtype ${DTYPE} > H100_${DEVICE}_${DTYPE}.log
DEVICE=cuda && DTYPE=int8 && python benchmark.py --device ${DEVICE}  --dtype ${DTYPE} > H100_${DEVICE}_${DTYPE}.log

| Device | Type | FP32 (TFLOPS) | BW | F16 | BF16 | INT8 |

@iakashpaul
iakashpaul / llm.sh
Last active February 17, 2024 15:50
LLMs on Android Via Termux >_
pkg install -y clang wget git
git clone --branch b2144 https://github.com/ggerganov/llama.cpp
cd llama.cpp/
make
cd models
@iakashpaul
iakashpaul / ref-bnb-code.md
Created October 20, 2023 09:37
For FP4+FP8 porting work
@iakashpaul
iakashpaul / run-llama2-with-less-than-8gb-VRAM-snippet.py
Last active August 19, 2023 10:59
run llama2 with less than 8gb VRAM snippet
# Below snippet lets you run Llama2 &B variants under 8GB VRAM comfortably, usually needing 5-7GB VRAM depending on configs
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
# run hugginface-cli login once after generating a read token after applying for the model weights from the same HF email id
model_name = 'meta-llama/Llama-2-7b-chat-hf'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_0 = AutoModelForCausalLM.from_pretrained(model_name, device_map=0,load_in_4bit=True,torch_dtype=torch.bfloat16)
MAX_LENGTH=100
@iakashpaul
iakashpaul / flan-ul2-snippet.py
Created August 18, 2023 13:12
Run Flan-UL2(20B) instances on V100s w/ 26GB VRAM per instance, change device_map according to your hardware. All layers are on GPU, additional GPU & CPU RAM specified only for reference
from accelerate import load_checkpoint_and_dispatch
from accelerate import init_empty_weights, infer_auto_device_map
from transformers import AutoConfig, AutoTokenizer, AutoModelForSeq2SeqLM, T5ForConditionalGeneration, T5Config
def load_model_sharded():
model_name="google/flan-ul2"
config = T5Config.from_pretrained(model_name)
tokenizer_1 = AutoTokenizer.from_pretrained(model_name)
max_memory_1={2: "30GiB",1: "10GiB", "cpu": "100GiB"}
@iakashpaul
iakashpaul / ffmpeg-audio-convert-snippet
Created April 13, 2021 07:38
for f in *.webm; do ffmpeg -i "$f" "${f%}.mp3"; done
@iakashpaul
iakashpaul / fix-missing-drives-aorus-x570-pro-wifi.md
Last active August 9, 2020 15:36

To fix missing NVMe/SATA drives on Gigabyte Aorus X570 PRO WiFi motherboards

TL;DR: Flash latest BIOS

After performing a fresh installation of Windows & Ubuntu on my PC, my motherboard stopped detecting the NVMe SSD to boot from.

This stemmed from the fact that I installed the latest chipset, WiFi/BT & Audio drivers from Gigabyte for Win 10, but didn't update the BIOS(in retrospect).

Clearing CMOS by shorting the 2 pins at the bottom-right of the motherboard didn't help, neither did loading optimized defaults for the BIOS. I just chanced to think that maybe updating the BIOS could help & poof! That did help with the issue. So pack an empty FAT/32 drive with the BIOS zip extracted & go into Q-Flash mode from the BIOS loader screen in order to flash it onto your motherboard.

@iakashpaul
iakashpaul / fix_exfat_drive.md
Created July 4, 2020 12:32 — forked from scottopell/fix_exfat_drive.md
Fix corrupted exFAT disk macOS/OSX

exFAT support on macOS seems to have some bugs because my external drives with exFAT formatting will randomly get corrupted.

Disk Utility is unable to repair this at first, but the fix is this:

  1. Use diskutil list to find the right drive id.
  2. You want the id under the IDENTIFIER column, it should look like disk1s1
  3. Run sudo fsck_exfat -d <id from above>. eg sudo fsck_exfat -d disk1s3
  4. -d is debug so you'll see all your files output as they're processed.
  5. Answer YES if it gives you the prompt Main boot region needs to be updated. Yes/No?
@iakashpaul
iakashpaul / bios-command.md
Last active April 3, 2020 11:52
Getting back into the BIOS after enabling Fast Boot on your motherboard (works for x570 Gigabyte Aorus Pro WiFi)

Fix

Because Fast Boot doesn't enable USB controllers right away on boot, you won't be able to enter the BIOS via the typical keyboard shortcuts like F10/F12/Del. The solutions are to reset the CMOS by removing the battery for 10 seconds, using a PS/2 keyboard(if you still have them or even the ports) or setting the jumper for the same according to your motherboard.

Thankfully instead of the above you can simply type the following to enter the BIOS via ubuntu, windows users have another shortcut of pressing shift & clicking the reboot button or something(google it).

systemctl reboot --firmware-setup