Read s1: what to measure when improving reasoning with little data

What is a model doing when its budget grows?

s1: Simple test-time scaling, published 2025-01-31, argues that enormous training runs are not the only route to stronger reasoning. The question is how existing models receive time to think during training and inference under limited compute. Unconditionally lengthening an answer is not the solution: irrelevant tokens cost money, and wrong stopping cuts off correct work.

1. 1Small set of high-quality reasoning examples
2. 2SFT
3. 3reasoning model

1. 1Problem
2. 2thinking budget
3. 3answer
4. 4verifier and evaluation

1. 1Increase or reduce budget
2. 2accuracy, token, and latency curve

The paper proposes selecting a small high-quality dataset for SFT and using budget forcing to control reasoning-token use. Read “small” carefully: separate dataset count from teacher-generation cost and base-model capability. Strong base models, curated examples, and particular math or competitive-programming evaluations are assumptions, not guarantees for a different workload.

Data selection is not simply deleting rows

The practical implication is that reasoning-data quality can dominate indiscriminate volume. Useful examples have a clear problem, verifiable final result, meaningful changes in approach, and little decoration. A long teacher trace is not automatically high quality: even a correct answer can include circular reasoning, calculation jumps, memorized material, or format violations. Record problem distribution, final-answer verifier, token length, language, duplicates, and teacher-generation conditions. Reading 20 examples by hand can expose a break sooner than filtering tens of thousands automatically, particularly for Japanese or internal documents.

Meaning and risk of budget forcing

At inference, budget forcing can be read as steering the model not to jump immediately to a final answer. Confirm implementation specifics in the paper and relevant official repository revision. Product design must address two failures: wasting time on easy problems, and letting users assume a long answer is correct.

1. 1Easy question
2. 2small budget
3. 3validator
4. 4finish quickly

1. 1Hard or uncertain
2. 2added budget
3. 3recheck
4. 4evidence-grounded answer

1. 1Budget ceiling
2. 2unknown or hold
3. 3person or follow-up job

Do not make a budget uniform. Solve first with a small allocation and add one only when a verifier fails, citations are insufficient, or a schema is broken. This controls waiting time and cost; it does not guarantee maximum accuracy. Repeating the same wrong answer after added attempts points to retrieval, tools, or ambiguous input rather than insufficient budget.

Reproduction design and operating rule

Read repository license, model terms, required GPUs, and data sources before use; record whether the paper version and repository commit correspond. Split 10–30 cases into design, validation, and locked final checking. Compare base model, SFT-only, and two-stage budget inference with fixed maximum tokens, sample count, temperature, and wall time. Measure accuracy, mean and p95 generation time, output tokens, interrupted runs, and JSON/citation failures. Label difficulty before running, then separate needless length on easy tasks from improvement on hard tasks.

Test-time scaling spends reasoning tokens, multiple samples, and verifier calls; long thought also increases the surface for content review. A small teacher dataset can be brittle outside its domain, and SFT does not necessarily find novel solutions. The useful exercise is to measure which failures another 1,000 tokens removes in your task. If the curve is flat, repair retrieval, input design, validation, or UI first.

Permit extra attempts only following test results—such as a broken schema, fewer than two retrieved sources, or failed calculation validation. After two attempts with the same failure, stop and hand off to a person or follow-up job. Retain both improved and unimproved problem IDs. Re-measure after model updates, and show users why a task takes time and that it can be stopped; do not expose inference parameters as product requirements.

Test whether added tokens change decisions

Budget forcing is useful only when extra computation changes a verifiable decision. On a fixed held-out set, store the first answer, final answer, stop reason, token count, and verifier result. Compute how often the final answer corrects a wrong first answer, how often it corrupts a correct one, and how often it merely repeats text. This distinguishes scaling from longer formatting.

A February 2025 Reddit thread about reproducing scaling on small models is an anecdotal warning: users reported both improvement and degradation as budget grew. Treat it as a reason to run the paired curve above, not as evidence for a threshold or a model-family rule.

Quiet-STaR (2024) is an earlier reminder that generating more reasoning carries a compute cost. For test-time scaling, predeclare the token or wall-clock cap and plot correctness against the cap on held-out tasks. Do not infer that an internal rationale is faithful merely because a longer budget improves a final answer.

PTTS (submitted 2026-09-23) compares independent branches with jointly planned outlines while keeping the executor fixed. Its benchmarks do not establish a general benefit for s1-like systems. Use it to add one diagnostic to a budget curve: count distinct approaches before execution, then plot verified completion and cost against that count; stop when added branches are duplicates.