Blog Post··4 min read
Data Efficiency in Pretraining: Packing, Batching, and What Gets Wasted
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.
Tag: sequence-packing
Blog Post··4 min read
Data Efficiency in Pretraining: Packing, Batching, and What Gets Wasted
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.