windows-install

gistfile1.txt

  Paste this in PowerShell:

  $env:MESH_LLM_INSTALL_FLAVOR = "cuda"
  irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

  Open a new PowerShell window if mesh-llm is not on PATH yet, then run:

  mesh-llm serve `
    --model Qwen2.5-3B-Instruct-Q4_K_M `
    --ctx-size 32768 `
    --auto

  Check it:

  curl http://localhost:9337/v1/models

  Console:

  http://localhost:3131

  Use Qwen2.5-3B-Instruct-Q4_K_M. That is in the current built-in recommended catalog, maps to Qwen/Qwen2.5-3B-Instruct-GGUF@main:q4_k_m, and is the
  right small CUDA-friendly choice for an RTX 4060 8 GB with 32k context.