The Gemma QA adapter, aligned by reinforcement learning against the shared reward model (0.983 held-out pairwise accuracy). PPO optimized the QLoRA policy toward higher reward while a KL penalty kept it anchored — the first run diverged and was re-run with a firmer KL anchor, which stayed bounded.

60 PPO iterations, ~36 min on an A100. Judged correctness 0.899 [0.878–0.918]; paired win-rate vs SFT 43% (ties split) — like DPO, RLAIF traded a little judged accuracy for alignment on Gemma. The reward model reached 0.983 held-out pairwise accuracy.

Architecture

Class	Gemma2ForCausalLM
Layers	26
Hidden size	2,304
Attention	8 heads / 4 KV · dim 256 · GQA
Feed-forward	GeGLU · inner 9,216
Attention window	sliding 4,096 (alternating) · logit soft-cap
Norm	RMSNorm
Context	8,192 tokens
Vocabulary	256,128
Embeddings	tied input/output

Training

Init from	gemma-2-2b QA-SFT adapter
Method	QLoRA — reward model + PPO
Trainable params	20.8M LoRA (0.8%)
Training data	500 preference pairs
Training tokens	335K per PPO pass
PPO iterations	60

What this model cost to build

$15.57 total Modal usage

our cost begins at fine-tuning — the base is Google's, imported free.

Stage	Detail	Cost
QLoRA QA fine-tune	the adapter RLAIF starts from	$9.84
Reward model	Bradley-Terry head on L4	$0.02
PPO alignment	60 iterations on A100	$4.23
Evaluation (shared)	13 versions on 500 held-out questions, this model's share	$1.48
Total		$15.57

Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.