#python
8 total
- Attention Residuals: Making Residual Connections Attention-Like
A reading of Kimi Team's Attention Residuals technical report: why residual connections should become attention-like too, and how Full AttnRes / Block AttnRes turn that idea into a trainable, deployable system
- Training Compute-Optimal Large Language Models: What Chinchilla Changed
The Chinchilla paper — why most large models were undertrained, and how to spend your compute budget wisely, with real Python code examples
- Scaling Laws for Neural Language Models: The Mathematics of Scale
The mathematics of scale — why bigger models are predictably better, with real Python code examples
- Language Models are Few-Shot Learners: GPT-3 and In-Context Learning
Larger models, better at eliciting abilities from context, with real Python code examples
- BERT: The Pre-Training Blueprint for Language Understanding
Establishing the pre-training paradigm, with real Python code examples
- Sequence to Sequence Learning: The Encoder-Decoder Blueprint
Establishing the encoder-decoder paradigm, with real Python code examples
- Neural Machine Translation by Jointly Learning to Align and Translate: Attention Before Transformers
The origin of attention mechanism, with real Python code examples
- Attention Is All You Need: The Transformer Blueprint
A study note on the Transformer paper, with real Python code examples