Skip to content

Instantly share code, notes, and snippets.

@mzbac

mzbac/gist:0f846e762ede4cb1b2372dc24ae4b264

Last active July 26, 2024 01:40
Show Gist options
  • Save mzbac/0f846e762ede4cb1b2372dc24ae4b264 to your computer and use it in GitHub Desktop.
Save mzbac/0f846e762ede4cb1b2372dc24ae4b264 to your computer and use it in GitHub Desktop.
Download ZIP
mxl-sharding
Raw
gistfile1.txt
install `pip install mlx-sharding`
For shard node:
Run `mlx-sharding-server --model mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx --start-layer 14 --end-layer 27`
For primary node:
Run `mlx-sharding-api --model mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx --start-layer 0 --end-layer 14 --llm-shard-addresses <your shard node address>`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment