Model · mistral-7b-v0-1
Mistral 7B (v0.1)
Developer: Mistral AI
Availability: available · checked 2026-09-24 · source
Raw record: /data/models/mistral-7b-v0-1.json
Fields
- id
- mistral-7b-v0-1
- identifiers
- huggingface
- mistralai/Mistral-7B-v0.1
- developer
- Mistral AI
- release_date
- 2023-09-27recorded · source
- weights_status
- open
- availability
- available · checked 2026-09-24 · source
- license
- Apache 2.0recorded · source
- architecture
- family
- decoder_only
- n_layers
- 32recorded · source
- hidden_size
- 4096recorded · source
- n_heads
- 32recorded · source
- vocab_size
- 32000recorded · source
- positional_encoding
- rotary (RoPE)recorded · sourcenote: Not named in the paper; from Mistral's release-day reference code.
- context_length
- 8192recorded · sourcenote: Paper Table 1 context_len. Config max_position_embeddings is 32768, a loader limit, not the described design value.
- n_kv_heads
- 8recorded · source
- sliding_window
- 4096recorded · source
- note
- Paper: 'Compared to Llama, it introduces a few changes' (GQA, sliding window, rolling buffer cache). Effective attention span per layer is the 4096 sliding window.
- normalization
- RMSNorm, pre-normrecorded · sourcenote: Not named in the paper.
- activation
- SiLU in a gated feed-forward unit (SwiGLU)recorded · sourcenote: Not named in the paper; w2(silu(w1(x)) * w3(x)).
- ffn_hidden_size
- 14336recorded · source
- training_data
- nullnot_recorded · sourcenote: Neither the release post nor the paper discloses the pretraining corpus.
- techniques
- transformer-decoderrotary-position-embeddingrmsnormswiglugrouped-query-attentionsliding-window-attention
- primary_sources
- record_history
- date:2026-09-24 · change:created from primary sources (Phase 1 seed, batch 1) · by:wilson-pruitt + claude ·date:2026-09-24 · change:context_length corrected 32768->8192 (paper); norm/activation/RoPE sourced to reference code; ffn width added (narrative/exercises/01-gpt2-xl-vs-mistral-7b.md) · by:wilson-pruitt + claude ·date:2026-09-24 · change:availability checked and recorded · by:wilson-pruitt + claude ·
Parents
No edges recorded.
Children
Weights descend
- ← fine_tuned_from Zephyr 7B β declared source Developer's own model card: 'fine-tuned version of mistralai/Mistral-7B-v0.1'.
- ← fine_tuned_from Mistral-7B-Instruct-v0.1 declared source Uploader (mistralai) is the developer of both this and the base model, so the base_model card claim counts as declared, not just declared_by_uploader.
- ← fine_tuned_from Mistral-7B-OpenOrca declared_by_uploader source Card: 'fine-tuned on top of Mistral 7B'.
- ← fine_tuned_from OpenHermes-2.5-Mistral-7B declared_by_uploader source HF card metadata base_model: mistralai/Mistral-7B-v0.1. Relation 'finetune' is the Hub's inference (tag), not stated in the card.
- ← fine_tuned_from SOLAR-10.7B-v1.0 declared source Card: built by 'depth up-scaling' (DUS): duplicating and interleaving Mistral 7B's layers into a taller 10.7B-parameter model (32->48 layers), then continuing pretraining on the WHOLE enlarged model (matches the doc's fine_tuned_from test: continued pretraining of the whole model, as with Code Llama from Llama 2 -- not a merged_from case, which requires no whole-model retrain). Panel-reviewed 2026-09-24, blind three-tier panel (Haiku/Sonnet/Opus, unanimous 3/3 for fine_tuned_from over an earlier solo depth_upscaled_from ruling that was reverted). All three panelists flagged the same open question: this loses the fact that the architecture was reshaped before training resumed, which this note is here to preserve.
- ← fine_tuned_from zephyr-7b-alpha declared_by_uploader source HF card metadata base_model: mistralai/Mistral-7B-v0.1. Relation 'finetune' is the Hub's inference (tag), not stated in the card.
- ← fine_tuned_from Mistral 7B SFT β declared source Card: 'This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the HuggingFaceH4/ultrachat_200k dataset.' Model description: 'Finetuned from model: mistralai/Mistral-7B-v0.1'. Uploader HuggingFaceH4 is the Zephyr developer.
- ← fine_tuned_from OpenChat 3.5 declared_by_uploader source No card sentence names the base. Evidence: config.json `_name_or_path: imone/Mistral_7B_with_EOT_token`, `model_type: mistral`, dims identical to Mistral 7B, card tag `mistral`; and the Starling-LM-7B-alpha card (a third party) says 'Openchat 3.5 (based on Mistral-7B-v0.1)'. Uploader imone is the OpenChat author, so this is the developer's own HF metadata, not a prose statement. The intermediate 'Mistral_7B_with_EOT_token' checkpoint (vocab +2) was not inspected.
Design
- ← same_architecture_retrained Mixtral-8x7B-v0.1 declared source Paper: 'Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks.' Pretrained from scratch -- no weights passed, so no fine_tuned_from edge.