K2-Horizon-MoVA-36B-A4B: open-source MoE model with 36B parameters and 4B active

The model IFM/K2-Horizon-MoVA-36B-A4B with the text-generation task has been published on HuggingFace. It is a sparse model from the K2-Horizon family: a Mixture-of-Experts architecture with Mixture-of-Values attention, storing 36 billion parameters while activating 4 billion per token. The author claims native support for a 512K token context and promises to release intermediate checkpoints, data, and training code.

What it means

The full model weights in the repository take up 69.7 GB in safetensors format, with the largest file being 3.9 GB. The license is apache-2.0, and it runs via transformers. For local use, 15 quantized builds are already available on the platform: GGUF, FP8, MLX, NVFP4, GPTQ. The most compact working build is IQ4_XS at 18.7 GB by NANI-Nithin. A model with 4 billion active parameters per token is a moderate load for running on your own hardware, especially in quantized form. The full weights will require a significant amount of VRAM, but quantized variants allow the model to fit on a single consumer GPU. The developer does not specify minimum memory requirements for different formats. You can keep up with new models and quantized builds in the CPU3D news feed.