Dataset · redpajama-data-1t
RedPajama-Data-1T
Builder: Together Computer
Availability: available · checked 2026-09-24 · source
Card lists 2084 jsonl files and download URLs; a 1B-token sample and recreation scripts are linked.
Raw record: /data/datasets/redpajama-data-1t.json
Fields
- id
- redpajama-data-1t
- identifiers
- huggingface
- togethercomputer/RedPajama-Data-1T
- builder
- Together Computer
- release_date
- 2023-04-17partial · sourcenote: HF dataset repo creation date, not the builder's own release announcement.
- availability
- available · checked 2026-09-24 · sourcenote: Card lists 2084 jsonl files and download URLs; a 1B-token sample and recreation scripts are linked.
- content
- Card: 'RedPajama is a clean-room, fully open-source implementation of the LLaMa dataset.' Token counts on the card: CommonCrawl 878B, C4 175B, GitHub 59B, ArXiv 28B, Wikipedia 24B, StackExchange 20B; total 1.2 trillion. Primarily English. The card says it 'was created to follow the LLaMa paper as closely as possible to try to reproduce its recipe.'recorded · sourcenote: The 'Data-1T' name and the 1.2T total on the card differ; both are the card's.
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (tranche 2 dataset records) · by:wilson-pruitt + claude ·
Parents
No edges recorded.
Children
Training data
- ← trained_on RedPajama-INCITE-7B-Base declared source Developer's card (Together is the uploader and developer): 'Training Data: Please refer to togethercomputer/RedPajama-Data-1T'; also in card metadata `datasets`.
Read in
No station on the reading path has touched this record yet.