Research

This study documents the design and evaluation of Antares, a scratch-trained decoder-only language model developed under a consumer-hardware wall-clock constraint. The final architecture was specified at 108.55M parameters with 16 Transformer blocks, width 768, grouped-query attention, SwiGLU feed-forward layers, RMSNorm, QK normalization, RoPE, tied embeddings, and a 16,384-token vocabulary. The development process included preliminary external-memory experiments, a short AdamW-versus-Muon optimizer comparison, base-model next-token probes, supervised post-training, and deployment analysis. On a synthetic associative-recall task, an external-memory hybrid improved matched-token accuracy relative to a Transformer baseline but reduced throughput, limiting its wall-clock advantage. In a short four-layer optimizer experiment, Muon reached a lower validation cross-entropy than AdamW at every recovered matched-token checkpoint, although the comparison was single-run and did not determine the optimizer used for the preserved final base checkpoint. Base-model probes identified strong lexical and selected factual associations together with pronounced prompt-direction asymmetry and weak calibration. Post-training datasets were effective at imposing chat, system-following, and tool-use formats, but a separate five-level reasoning-mode dataset was rejected as evidence of reasoning improvement because of substantial template repetition. In a full zero-shot causal-likelihood evaluation, SFT improved the five-task mean from 39.15% to 42.37%; Antares v0 reached 42.37% versus 42.47% for GPT-2 Small while using 12.8% fewer parameters. Inspection of the serving runtime also showed that calculator use and several freshness-sensitive web queries are deterministically routed outside the neural model, motivating explicit separation of model capability from orchestration behavior in evaluation.

We study what can be learned about small-language-model architecture when the scarce resource is not parameter count or nominal FLOPs, but wall-clock research time on one consumer GPU. Horizon is a 20.06M-parameter decoder-only Transformer trained from scratch with a 16,384-token vocabulary on the same 2.495B-token base corpus used by Antares. An initial 2B-token saturation protocol was superseded before the main architecture comparison after a measured synthetic ceiling of 206,233 tokens/s showed that 2B tokens could not fit inside one hour even before evaluation or checkpoint overhead. We therefore adopted a hard 3,600-second principal GPU-active budget and treated architecture research as an adaptive allocation problem. Compute-matched recurrent weight sharing reduced parameters from 20.06M to as little as 8.01M but monotonically worsened validation loss and did not improve end-to-end throughput. A parameter-matched 160-dimensional factorized lexical embedding doubled perplexity at 25M tokens. External memory produced the only raw loss improvements over dense D8 at 25M, but the gains were tiny and came with 2.18x and 6.17x wall-clock penalties. The remaining budget was consequently allocated to continuing dense D8 from 25M to 200M tokens. Validation perplexity fell from 218.60 at 25M to 19.80 at 200M while the entire eight-run principal program consumed 57.53 GPU-active minutes and 147.83 Wh of measured GPU energy. Direct next-token probes show that this smooth perplexity improvement did not imply robust factual recall: at 200M tokens, “Paris” ranked only 271st after “The capital of France is”. The study therefore supports a systems-aware conclusion: under a strict consumer-GPU wall-clock constraint, simple dense capacity plus additional tokens dominated the tested architectural compression and memory mechanisms.

Zeon marked the transition from notebook-scale language-model experiments to a publicly deployed EXOFORCE AI model family. This report reconstructs Zeon-1 and Zeon-2 from preserved code, dataset metadata, server logs, SFT progress files, and a public launch post dated 29 April 2026. Zeon-1 used a 12-layer, 512-wide causal Transformer with 54,800,896 parameters and an 8,192-token tokenizer trained over a FineWeb-derived corpus. The preserved Colab run reached at least 26.5 million training-token presentations before interruption, while production logs confirm approximately one token per second on the project server. Zeon-2 retained the same deployed ChatLM architecture and concentrated development effort on assistant-only SFT and learned calculator/time tool routing. A 600-step preview was followed by a 4,000-step final SFT run recording 3,698,586 supervised tokens. A separate RoPE/GQA Zeon-2 pretraining implementation was also recovered, but its state layout is incompatible with the deployed Zeon-2 runtime and is therefore treated as an alternate experimental branch rather than the production model. The resulting record is useful because it separates model architecture, training intent, deployed behavior, and runtime tool execution without retroactively attributing capabilities to the wrong component.

This retrospective traces a two-year progression from early recurrent conversational models to Antares v0, a 108,552,960-parameter scratch-trained model that reaches a 42.37% mean on a five-task zero-shot likelihood benchmark, compared with 42.47% for GPT-2 Small at 124,439,808 parameters. The path was not monotonic in parameter count. MICRO-3 used 167,882,301 parameters yet showed strong training-corpus reuse in public generation. Miranda experiments explored both LSTM and encoder-decoder Transformer designs, including a public 6.36M variant intended for fast CPU inference. Zeon standardized a 54.8M causal decoder-only stack, expanded web-text pretraining, deployed the model publicly, and added learned calculator/time tool routing through SFT. Antares then increased scale to 108.6M while adopting a more deliberate evidence and evaluation process. The main lesson is that useful progress came from better objectives, broader data, architecture alignment, post-training design, and measurement—not from parameter growth alone.

Miranda and MirandaGPT were a cluster of 2024 language-model experiments rather than a single immutable architecture. Preserved notebooks show a 50.61M-parameter LSTM model named Miranda, a larger encoder-decoder Transformer experiment named MirandaGPT, and a later Kaggle MirandaGPT configuration trained on the same human-conversation corpus used by other early models. A contemporaneous public announcement from 16 August 2024 separately describes a 6,365,070-parameter MirandaGPT that ran quickly on CPU and produced short expository completions. The name reuse means these artifacts should be treated as iterations, not forced into a single checkpoint lineage. This report documents the architectures, training data, preserved generations, and implementation limitations that affect interpretation. The main historical result is constructive: compared with the earlier MICRO series, Miranda work shifted attention from raw model size toward smaller deployable models, cleaner generation behavior, and Transformer-based experimentation, setting up the later transition to causal decoder-only models.

The MICRO series was an early attempt to train conversational language models from scratch using recurrent architectures and a compact dialogue corpus. Preserved notebooks establish a 20.71M-parameter MICRO-1 and a substantially larger 167,882,301-parameter MICRO-3. MICRO-3 used 4,090-dimensional embeddings and a 4,090-unit LSTM over 300-token sequences; a contemporaneous post already described it as somewhat overfit. Comparing its preserved public generation with the recovered human_chat corpus confirms substantial near-verbatim reuse of training conversations. A later MICRO-4 notebook explored a much smaller recurrent configuration but used an objective that is not directly comparable to next-token language modeling. The series is historically important because it demonstrates, in unusually clear form, that parameter count alone did not determine usefulness. Later EXOFORCE AI models became smaller, better aligned with causal language modeling, and more deliberate about data and evaluation.

The earliest preserved project archive is not a sequence of finished language models; it is a progression of small experiments used to learn data preparation, tokenization, neural sequence modeling, generation, and deployment. This report documents selected 2023-2024 artifacts that can be supported directly by surviving notebooks. A March 2024 notebook titled TinyLanguageModel is actually an inference experiment with the public Aya-101 model and is therefore not counted as a scratch-trained project model. By June 2024, LightGPT and a larger GPT notebook implement custom encoder-decoder Transformers. The GPT notebook trains a 20,977,133-parameter model over 99,951 Cornell Movie-Dialogs samples and preserves 22 complete epochs before a partial twenty-third epoch. Other short-lived notebooks exercise Transformer classes or synthetic data without forming release candidates. The value of this period is methodological: it established the code and conceptual foundation from which the later own-model lineage emerged.