Qwen-14B (Alibaba) The most powerful open-source model for it's size. And the longest trained: 3T tokens. Comes in 5 different versions: Base, Chat, Code, Math and Vision. (And is even trained for tool usage!) Opinion: You should consider it as your new "go-to". --- Paper: qianwen-res.oss-cn-beijing.a… Qwen-14B: huggingface.co/Qwen/Qwen-14B Qwen-14B-Chat: huggingface.co/Qwen/Qwen-14B… Qwen-VL-Chat: huggingface.co/Qwen/Qwen-VL-… --- --- Implementation Several details caught my eye while reading it's paper. These might be interesting for anyone involved with training models at this scale. --- Tokenizer Like many other multilingual LLMs, they had to add language specific tokens to their tokenizer. - Used GPT-4's tokenizer (CL100k) - Spllited numbers into single digits. - Added Chinese common words. - The final tokenizer has a vocab of 152K. - Result: The compressive rate of this model is better than all the competing models. --- Pretraining Used a diverse set that covers a wide range of topics from various sources, including: Web documents, encyclopedia, books, codes, etc. - The actual sources are not listed. - They do mention that they extract text from HTML pages and use language identification tools to determine the language. (Hinting: Mass scraping) Deduplication methods: - Exact-match. - MinHash. - LSH. Filtering methods: 1. Rule based. 2. ML based: - ML models train to estimate: Text-quality (scoring), Identify inappropriate content. By hand: - They manually sample texts from various sources, review them by hand to ensure high quality. - They selectively up-sampled specific datasets coming from certain sources. Hyperparameters: - AdamW: betas = (0.9, 0.95), eps = 1e-8, cosine scheduler, min_lr = 10% of max_lr (Aggressive choice!), BF16. --- Supervised Fine-Tuning As said, they further fine-tune the base model for specific domains. In the process, they use: - Self-instruct for generating synthetic high quality data. - For evaluation, they tested for code excitability (Running the generated code). - There is also a usage example demonstrating the ability to "Auto Comment" a code snippet, on this topic: In my own opinion, this is one of the "secrets" of ChatGPT: Generating comments BEFORE each line of code and use the commented code as a training data. You can easily see that all the code ChatGPT generates follow this structure and it is obvious why this type of format boosts coding performance. --- Architecture Tweaks RoPE: Like everyone, they use RoPE (One of the best options today) but they are using FP32 precision for the inverse frequency matrix, rather than BF16 or FP16. > Noted. [1] RoPE is (VERY) powerful: blog.eleuther.ai/rotary-embe… Bias: Again like everyone, they removed all biases [1] nearly everywhere BUT they added biases for the QKV layers [2]. [1] Bias removal improves performance: arxiv.org/abs/2204.02311 [2] RoPE + Bias extrapolation study: spaces.ac.cn/archives/9577 Normalization: RMSNorm. Pre-Norm. Nothing new under the sun but they do hint that better methods exist today other than pre-norm and will be experimented with in the future. They might be hinting at this: github.com/ZixuanJiang/pre-r… (?) --- Context window length - Using dynamic NTK-aware interpolation during inference. [1] - Specifically, they used YARN's version: dynamic interpolation version that changes the scale by chunks. [2] - Interesting observation: Long-context modeling ability varies across layers - Lower layers being more sensitive in context length extension compared to the higher layers. To leverage this: They shorter windows for lower layers and longer windows for higher layers. [1] Dynamic NTK: reddit.com/r/LocalLLaMA/comm… [2] YARN: arxiv.org/abs/2306.15595 --- Synthetic Data Generation Self-instruct: They used the few-shot capabilities of the model to generate data [1], But they do this in an iterative way: At each generation step, they manually remove noisy samples and retrain the model. Doing this process multiple times resulted in a much more reliable model for fewshot generation. [2] [1] - Giving it couple of examples and let it "complete" the text to generate more. [2] - This is probably because of the main issue with synthetic data: Filtering. If you do not filter the data well enough you amplify the noise and not just the signal. --- RLHF The paper also detail heavily the process of RLHF training. This is less of an interest of myself at the moment but I highly suggest reading it if you are interested. --- Summary These models are unbelievably powerful and probably should be used as your new "go-to" models. (For at least code and math..) Amazing work!