How to choose a model + quantisation for 128GB of unified memory, community edition.
- Start from the model roster & benchmarks — measured on this hardware class, not cloud GPUs.
- Quant picks: q4_k_m is the usual sweet spot; q8_0 when quality matters and memory allows; below q4, judge quality yourself per model.
- MoE models punch above their weight here — active-parameter count drives speed, total parameters drive memory.
- Leave headroom: the OS + other slots + KV cache all share the pool. A model that “fits” with 2GB spare will hurt at long context.
Wiki — add model-specific notes (chat template quirks, tool-calling reliability, vision support).