Evaluation
Training produces many checkpoints. Evaluation is how you decide which one to fine-tune from, and which one to ship.
Both tools live inside the training sections rather than as separate areas of the product: Pre-training and Fine-tuning each have their own Leaderboard tab, and benchmarks are created from and consumed by those leaderboards.
Leaderboard
The leaderboard ranks checkpoints across runs, so you can compare a checkpoint from one run against a checkpoint from another under the same measurement.
Two properties matter more than the ranking itself:
- It is scoped to its section. The Pre-training leaderboard ranks Pre-training runs; the Fine-tuning leaderboard ranks fine-tuning runs. The two are optimizing for different objectives and are not comparable, which is why they are not mixed.
- It ranks checkpoints, not runs. A run's best checkpoint is often not its last one. Selecting the final checkpoint by default is the single most common way to ship a worse model than you trained.
Use it to pick the base checkpoint for a fine-tuning run, and to pick the checkpoint you deploy.
Benchmarks
A benchmark fixes the thing you measure against, so that a comparison means something. Without one, two checkpoints evaluated on different data are two unrelated numbers.
You add checkpoints to a benchmark from the leaderboard, and the benchmark's results come back into the leaderboard as a comparable column.
Metrics
Results are reported against the metrics your Model Design enabled:
| Metric | Reads as |
|---|---|
accuracy | Share of correct predictions |
precision | Of the positives predicted, how many were right |
recall | Of the actual positives, how many were found |
f1 | Harmonic mean of precision and recall |
roc_auc | Ranking quality across all thresholds |
log_loss | Penalty on confident wrong answers |
ks | Maximum separation between the two distributions |
gini | Discriminatory power, rescaled from roc_auc |
Custom metrics defined in the design appear alongside these.
Metrics arrive from four sources, and the distinction matters when a number looks wrong:
TRAIN (emitted during training), CHECKPOINT_LOSS and CHECKPOINT_VALIDATION (evaluated
per checkpoint against the design's test split), and BENCHMARK (evaluated against a
benchmark's fixed dataset).
Reading a comparison honestly
- Compare on the metric the task is judged on, not on whichever metric moved most.
- Hold the benchmark fixed. A metric that improved against different data has not improved.
- Watch the validation curve, not just the final number. A checkpoint whose validation loss has started rising is memorizing, and the leaderboard's headline metric can still look fine.
Next steps
- Pre-training · Fine-tuning
- Deploy a model
- Drift Radar — evaluation does not stop at deployment.