MoDE is a release of the Mixture of Decoupled Experts experiment: train or collect dense experts, train lightweight supervised routers over a frozen backbone, then assemble a strict top-1 MoE checkpoint.
If you're looking for a demonstration of the MoDE technology, check out the Hugging Face release:
https://huggingface.co/SL-AI/GRaPE-2-Ultra
| Path | Purpose |
|---|---|
MoDE_RESEARCH_PAPER.md |
Technical report |
train_mode_global_router.py |
Global router trainer |
train_mode_per_layer.py |
Per-layer router trainer |
train_mode_per_layer_smart.py |
Smart per-layer router trainer |
assemble_mode_model.py |
Assembles dense experts plus router into MoDE |
data/ |
1k public inspection sample |
demos/ |
A few demonstrations of GRaPE 2 Ultra |
The scripts use local placeholders:
models/base-general
models/expert-creative
models/expert-code
models/expert-thinking
models/expert-auxiliary
data/mode_dataset.parquet
outputs/
Set those paths to your own local checkpoints and data. Model-loading scripts default to local/offline loading so they do not hang while trying Hugging Face metadata requests. To intentionally load a remote model ID, run with:
MODE_ALLOW_REMOTE_MODEL_LOAD=1 python train_mode_per_layer_smart.pyThe full private training corpus is not included. The release sample is for schema inspection only and is not large enough to train a router.