Skip to content

Instantly share code, notes, and snippets.

@wassname
Last active March 1, 2025 00:31
Show Gist options
  • Save wassname/0e38060a8529653b4d0675d00b61228a to your computer and use it in GitHub Desktop.
Save wassname/0e38060a8529653b4d0675d00b61228a to your computer and use it in GitHub Desktop.
When a peft adapter has an invalid base model, how do I fix it?
peft_model_id = 'v2ray/GPT4chan-8B-QLoRA'
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoConfig, LlamaConfig
from peft import PeftConfig, PeftModelForCausalLM
peft_config = PeftConfig.from_pretrained(peft_model_id)
# This model points to a local base model, which we don't have. So lets redirect it to a public version
peft_config.base_model_name_or_path="unsloth/Llama-3.1-8B"
# now load the modified config in 8bit
base_model = AutoModelForCausalLM.from_pretrained(
peft_config.base_model_name_or_path,
quantization_config=BitsAndBytesConfig(load_in_8bit=True)
)
# if the peft comes with an expanded embedding space
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
base_model.resize_token_embeddings(len(tokenizer))
# and add the adapter
model = PeftModelForCausalLM.from_pretrained(
base_model,
peft_model_id,
config=peft_config)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment