Kolibri Pre-Training from Scratch: UniBPE, Hybrid Attention and Exact Quantile Balancing
Building Aleph Alpha's Kolibri base model from the ground up: the Unigram-loss merge score of UniBPE and why it is pointwise mutual information in disguise, the KV-cache arithmetic of 40 sliding-window and 10 NoPE blocks, sigmoid MoE routing with Exact Quantile Balancing and Load-Error Injection, the 1 − K/E signature of a router the bias has overridden, and the cumulative-learning-rate law behind hyperparameter transfer. Every step is derived on one German sentence and four tokens.