Model · falcon-7b
Falcon 7B
Developer: Technology Innovation Institute (TII)
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/falcon-7b.json
Fields
- id
- falcon-7b
- identifiers
- huggingface
- tiiuae/falcon-7b
- developer
- Technology Innovation Institute (TII)
- release_date
- 2023-04-24partial · sourcenote: Creation date of the developer's HF repository (may predate public release). Card: training happened in early March 2023.
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- Apache 2.0recorded · source
- architecture
- family
- decoder_only
- note
- Card: 'broadly adapted from the GPT-3 paper' with rotary embeddings, multiquery attention + FlashAttention, and parallel attention/MLP with a single layer norm. d_model raised to 4544 'to compensate for multiquery'; head_dim 64.
- n_layers
- 32recorded · source
- hidden_size
- 4544recorded · source
- n_heads
- 71recorded · source
- n_kv_heads
- 1recorded · sourcenote: Config multi_query: true (one shared key/value head).
- vocab_size
- 65024recorded · source
- positional_encoding
- rotary (RoPE)recorded · source
- context_length
- 2048recorded · source
- training_data
- 1,500B tokens: RefinedWeb (filtered, deduplicated web crawl; 79% English) plus curated corpora, some 'inspired by The Pile'.recorded · source
- techniques
- transformer-decoderrotary-position-embeddingmulti-query-attentionparallel-attention-ffn
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (Phase 1 seed, batch 2) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
Training data
- trained_on → RefinedWeb declared source RefinedWeb-English is 79% of 1,500B training tokens; the rest is curated corpora (not recorded as datasets).
Design
- design_follows → GPT-3 (175B, 2020 paper) declared source Card: 'broadly adapted from the GPT-3 paper'.
Children
Weights descend
- ← fine_tuned_from falcon-7b-instruct declared source Card: 'built by TII based on Falcon-7B and finetuned on a mixture of chat/instruct datasets.'
Read in
No station on the reading path has touched this record yet.