Zhang Peiyuan

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

Some preliminary explorations of Mamba's context scaling.

No project description available.

Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.

Attn-QAT: 4-Bit Attention With Quantization-Aware Training