🏡/repos/ryanssenn/

torchless

🔥

Torchless

LLM inference engine built from scratch in C++

A custom-built inference engine that runs Mistral 7B on CPU without PyTorch or any ML frameworks. Hand-coded transformer implementation with full tokenizer, quantization, and inference pipeline.

~1200 lines C++CPU-onlyINT8 quantization
✨

Why Torchless?

Understanding LLMs from first principles

Zero Dependencies

No PyTorch, TensorFlow, or ONNX. Every operation hand-coded in C++ with OpenMP for parallelization.

Educational Focus

Clear, readable implementation of every transformer component. Perfect for understanding how LLMs actually work.

Full Pipeline

Complete from model export to text generation. Includes tokenizer, quantization, and inference.

🏗️

Architecture

End-to-end inference pipeline

📦

Model Export

HF → Binary

📝

Tokenizer

BPE encoding

🧠

Transformer

32 layers

🎲

Sampling

Token generation

Mistral 7B
Target model
32 layers
Transformer depth
4096 dim
Hidden size
8 KV heads
GQA groups
📦

Model Export Pipeline

Converting Hugging Face models to binary

Export Process

  • • Load model from Hugging Face safetensors
  • • Extract config, vocabulary, and BPE merges
  • • Apply optional INT8 quantization
  • • Package into single binary file
  • • Memory-mapped loading for efficiency

Binary Format

[8-byte header size]
[JSON header]
  - config (dims, layers, heads)
  - vocab (token → id mapping)
  - merges (BPE merge rules)
  - tensor offsets
[Padded tensor payload]
  - weights (f32 or int8)
  - quantization scales

Quantization Strategy

Per-Group Symmetric Quantization

  • • Group size: 64 values
  • • Scale computed per group
  • • Maps to INT8 range [-127, 127]
  • • On-the-fly dequantization during matmul

Memory Savings

  • • Float32: 4 bytes/param
  • • INT8: ~1 byte/param + scales
  • • ~4x reduction in model size
  • • Faster memory bandwidth
📝

Tokenization

Byte-Pair Encoding from scratch

BPE Implementation

  • • Full BPE encoder/decoder
  • • Loads vocab and merge rules from tokenizer.json
  • • Mistral-specific pre-tokenization (Metaspace)
  • • Byte fallback for unknown characters
  • • ~209 lines of implementation

Tokenization Flow

1Split text by whitespace/punctuation
2Convert each word to bytes
3Apply BPE merges iteratively
4Map tokens to vocabulary IDs
🧠

Transformer Implementation

Every layer hand-coded in C++

Embedding

~15 lines

Token ID lookup in embedding table. Maps 32K vocab to 4096-dim vectors.

RMSNorm

~20 lines

Root Mean Square normalization. Simpler and faster than LayerNorm.

RoPE

~35 lines

Rotary Position Embeddings. Encodes position through rotation in complex plane.

Attention

~60 lines

Multi-head Grouped-Query Attention with KV cache. 32 query heads, 8 KV heads.

MLP

~25 lines

SwiGLU feedforward: gate * silu(up) projection with 14336 intermediate dim.

LM Head

~10 lines

Project final hidden state to vocabulary logits for next token prediction.

Grouped-Query Attention (GQA)

How GQA Works

  • • 32 query heads with 128-dim each
  • • 8 key-value head pairs (shared)
  • • Reuse factor: 4 query heads per KV pair
  • • Reduces KV cache memory by 4x

KV Cache Optimization

  • • Stores K/V across sequence positions
  • • Avoids recalculating for past tokens
  • • Critical for efficient generation
  • • Managed in InferenceState struct
⚡

CPU Kernels

Optimized math operations

matmul

Matrix multiplication with quantization support

row_matmul

Row-wise matmul for attention scores

softmax

Temperature-aware with numerical stability

rope

Rotary embeddings application

silu

SiLU activation: x * sigmoid(x)

element_ops

Add, multiply, power, sqrt

Optimization Features

OpenMP parallelizationLoop unrollingCache-friendly access patternsOn-the-fly dequantization
🗂️

Inference State Management

Pre-allocated memory for zero-copy inference

InferenceState Structure

  • • Hidden states and residuals
  • • Query/Key/Value projections
  • • Attention scores and context
  • • MLP intermediate activations
  • • Output logits and probabilities
  • • KV cache per layer

Memory Management

  • • Arena allocator (256 MB)
  • • Zero-copy tensor views
  • • Memory-mapped model file
  • • Single token processing (batch=1)
  • • Reused buffers across tokens
🔄

Inference Lifecycle

How text generation works

1
Loading
Memory-map binary file, parse JSON header, create tensor views
2
Tokenization
Convert input text to token IDs using BPE encoding
3
Transformer Forward
Process through 32 layers: Embed → (RMSNorm → Attention → MLP) × 32 → Final Norm → LM Head
4
Sampling
Apply softmax to logits, sample next token (greedy or multinomial)
5
Auto-regressive
Append token, update KV cache, repeat from step 3
🔬

Single Transformer Layer

Inside one of 32 layers

Input Hidden State
RMSNorm (pre-attention)
Q/K/V Projection → RoPE → Attention → Output Proj
Residual Add
RMSNorm (pre-MLP)
Gate↑ × SiLU(Up↑) → Down↓ (SwiGLU MLP)
Residual Add → Output Hidden State
💻

Usage

Building and running

Export Model

python3 export_mistral.py \
  --model_dir ../Mistral-7B-v0.1 \
  --out ./mistral.bin \
  --quant f32  # or int8

Build & Run

# Compile
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .

# Run inference
./torchless ../mistral.bin "Paris is the capital of"
📁

Code Structure

Source Files

src/tokenizer/BPE tokenizer (209 lines)
src/model/mistral/Transformer modules (181 lines)
src/backend/cpu/Math kernels (154 lines)
src/loader/Model loading (157 lines)
src/common/Tensor infrastructure (107 lines)
main.cppInference entry point

Python Scripts

export_mistral.pyHF to binary converter
parity/Component validation tests

Build

CMakeLists.txtCMake configuration
requirements.txtPython dependencies
🚀

Development Focus

Current Work

  • • Performance optimization
  • • CPU SIMD vectorization
  • • Custom CUDA kernels
  • • Mistral 3 architecture support

Planned Features

  • • Temperature scaling for sampling
  • • CPU multithreading optimization
  • • Chat/conversation interface
  • • Ministral 3 3B support
💡

Key Takeaways

Framework-free inference

Demonstrates that ML frameworks are conveniences, not requirements. Every operation can be implemented directly.

Zero-copy architecture

Memory-mapped files and tensor views eliminate data copying. Model weights accessed directly from disk.

Quantization from scratch

Per-group INT8 quantization with on-the-fly dequantization. Understand exactly how model compression works.

Educational codebase

~1200 lines of readable C++ covering the complete inference pipeline. Great for learning transformer internals.