Posts tagged "tokenizers"
2 posts found
Kolibri Pre-Training from Scratch: UniBPE, Hybrid Attention and Exact Quantile Balancing
Building Aleph Alpha's Kolibri base model from the ground up: the Unigram-loss merge score of UniBPE and why it is pointwise mutual information in disguise, the KV-cache arithmetic of 40 sliding-window and 10 NoPE blocks, sigmoid MoE routing with Exact Quantile Balancing and Load-Error Injection, the 1 − K/E signature of a router the bias has overridden, and the cumulative-learning-rate law behind hyperparameter transfer. Every step is derived on one German sentence and four tokens.
X-Token: Cross-Tokenizer Knowledge Distillation from Scratch
Building cross-tokenizer distillation from the ground up — why per-position KL breaks across tokenizers, DP span alignment, the chain-rule chunk merge, the projection matrix W, and two complementary losses (P-KL and H-KL) — with a full proof of GOLD's suppressive gradient and one running 2+3 example derived step by step.