AI Research Lab · Project
Architecture Comparison
Choosing architectures based on evidence, not hype.
Overview
A rigorous, apples-to-apples study of how different sequence-model architectures behave when trained under identical conditions on our in-house benchmarks — so our design decisions are grounded in measured results rather than trends.
Objectives
- Fair comparisonIdentical data, compute, and evaluation across models.
- Efficiency vs. qualityMap the tradeoffs for each architecture.
- Scaling behaviorUnderstand how models improve with size and data.
- ReproducibilityTransparent, repeatable experiments.
Our Approach
- Controlled trainingStandardized budgets, data, and hyperparameters.
- Diverse candidatesGPT-2, Llama-style, GQA, Mamba, and RWKV — all pure-PyTorch.
- Unified harnessOne consistent evaluation pipeline for all runs.
- Open methodologyDocumented protocols and results.
Focus & Tech
GPT-2LlamaGQAMambaRWKVTinyStories
Status: Harness open-sourced · runs in progress
Interested in this work?
We collaborate on architecture research and evaluation.