JeanKaddour/minipile

Dataset Card for MiniPile Dataset Description The MiniPile Challenge for Data-Efficient Language Models Dataset Summary MiniPile is a 6GB subset of the deduplicated The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using k-means, and (3) filter out low-quality clusters. The primary motivation for curating MiniPile is that (i) diverse…