A small proxy run justifies a full training budget only when it is one point on a fitted curve. On its own it proves nothing. The defensible method has three parts, plus a fourth for mixture-of-experts models: fit a Chinchilla-style loss-versus-compute curve across several proxy sizes, test that fit against a held-out proxy, and promote only when the predicted gain at full compute clears a numeric tolerance set from the fit’s own residuals and confidence intervals. For mixture-of-experts (MoE) models, the fourth part is to add the variables dense laws leave out and to give throughput a curve of its own.
LynxBenchAI measures throughput, latency and memory on your own hardware, so the wall-clock cost of a full run does not rest on a FLOPs estimate alone; the first step is to benchmark the hardware the full run will use.
TechnoLynx builds this proxy-to-full-scale gate for training teams. The first step is to frame the target budget, the proxy scale range and the metrics to track. After that comes the size sweep, then fitting and backtesting the curves on the team’s hardware, and finally a promotion threshold the team can run as a gating step.
How many proxy sizes before you trust a Chinchilla-style extrapolation?
You need at least three proxy sizes to determine a three-parameter curve, and a fourth to test it. One run cannot validate an extrapolation. The reason is arithmetic: a power law with an irreducible loss term, L(C) = aC^b + c, has three free parameters, so three points only determine it, and you need a fourth before there is any residual left to check.
Size range matters as much as the count. Hoffmann et al. (2022) observed curvature in the FLOP-minimal loss frontier, which means projections from very small models give different predictions from projections built on larger ones. Their study trained over 400 models from 70 million to over 16 billion parameters, and most of them had more than 500 million parameters, unlike Kaplan et al., where many runs were below 100M.
Two further findings shape the sweep. Choshen, Zhang and Andreas (2024) found that fitting to intermediate checkpoints, not just final losses, substantially improves accuracy. They also found that seed variance can make several small models more useful than one larger one. Gadre et al. (2024) set a lower boundary: when the token multiplier is too small (they flag 5 tokens per parameter), scaling becomes unreliable and falls off the trend. Our working rule follows from these results. Use at least three sizes plus a held-out fourth, push the largest proxy as close to the target as the budget allows, run extra seeds at the small end where noise is worst, and keep every run above the under-training boundary.
One result cuts the other way and deserves a fair hearing. The DataDecide study (2025) found that ranking candidates at a single small scale predicted the winner at 1B parameters roughly 80% of the time, and none of the eight scaling-law methods it tested beat that single-scale baseline for the compute spent. The same authors point out that a single-scale ranking cannot detect a crossover between the small experiments and the target, and that the trends they observed crossed over frequently. A single-scale ranking is cheap and strong when nothing crosses. A multi-point fit is the only thing that tells you whether something will.
The fitting procedure, step by step
The Chinchilla paper describes three ways to fit the frontier. Running all three on your own proxies gives you a built-in consistency check.
- Train each size with schedule-matched learning rates. Hoffmann et al. trained four models per parameter count, with learning-rate decay horizons ranging by a factor of 16 in tokens. They found that matching the cosine schedule to the number of training tokens gives the best final loss whatever the model size. Intermediate losses taken from a longer fixed schedule overestimate loss, and that bias contributed to the earlier conclusion that model size should grow faster than data.
- Count total parameters and estimate compute at roughly 6N FLOPs per token. Kaplan et al. (2020) estimate non-embedding training compute at about 6N floating-point operations per token, counting the backward pass. Pearce and Song (2024) recommend that scaling studies use total parameters and total compute.
- Build the frontier (Approach 1). Smooth and interpolate each run’s loss curve to get a continuous FLOP-to-loss mapping, then at each FLOP value take the run with the lowest loss. Hoffmann et al. did this at 1,500 log-spaced FLOP values.
- Fit the allocation exponents. Fit N_opt ∝ C^a and D_opt ∝ C^b to that frontier. Approach 1 gave a = b = 0.50.
- Cross-check with IsoFLOP profiles (Approach 2). Fit a parabola to loss against model size at each fixed compute budget and read off the minimum. This gave a = 0.49 and b = 0.51.
- Cross-check with a parametric loss (Approach 3). Fit L(N, D) = E + A/N^α + B/D^β directly. This gave a = 0.46 and b = 0.54.
- Compare the three. In the Chinchilla study the three approaches gave comparable predictions. If they disagree on your proxies, the size of that disagreement is the first number to report upward, ahead of any extrapolated loss.
Chinchilla or Kaplan: which law should a proxy fit start from?
Start from the Chinchilla form fitted on total parameters and total compute. Kaplan’s exponents are a historical result whose bias is now explained. Kaplan et al. (2020) found that optimal model size grows as N ∝ C^0.73, so most extra compute should go into parameters. Hoffmann et al. (2022) found N ∝ C^0.50, which means model size and training tokens should grow in equal proportion.
Pearce and Song (2024) attribute much of the gap to Kaplan counting non-embedding rather than total parameters while working at small scale. When they simulated the Chinchilla study under those conditions, they got biased coefficients close to Kaplan’s. Porian et al. (2024) reproduced the Kaplan law and named three more factors: last-layer computational cost, warmup duration and scale-dependent optimiser tuning. Once those were corrected, their results agreed closely with Chinchilla. Hoffmann et al. also tested the two laws head to head. Kaplan’s law pointed to 4.68 billion parameters and theirs to 2.86 billion, and the model trained to their prediction performed better.
Every one of those corrections is something a proxy sweep can get wrong on its own: how parameters are counted, how warmup is set, whether the learning rate is retuned at each size. Kaplan’s paper still has a use. It shows that training curves follow power laws you can extrapolate from their early part to roughly predict the loss of a longer run. Its authors were also candid that without a theory of the corrections to their laws, “it’s difficult to determine in what circumstances they can be trusted”.
How wide the uncertainty on the Chinchilla scaling laws really is
Even the reference law carries wide error bars, so a proxy pipeline that reports a single point estimate is understating its risk. Besiroglu and Erdil (2024) partially reconstructed the Chinchilla data. They found that the Approach 3 parameters as rounded in the paper fit the data poorly, while their own refit left residuals tightly concentrated around zero. Their 80% confidence interval for the parameter exponent is roughly 50 times wider than the one Hoffmann et al. reported. At 1e26 FLOP and above, their fit is consistent with anywhere between 4 and 40 tokens per parameter, around a point estimate of 25.6. That is a tenfold spread in how to split the same budget.
The noise is also uneven across the sweep. Hoffmann et al. note that points below about 1e21 FLOPs have larger residuals, and their Huber-loss fit automatically treats those low-compute points as outliers and gives them less weight. They also had only two comparable large-scale runs to check against, and they observed concavity at high compute budgets that suggests the power law may overestimate the optimal size of large models. In the Chinchilla fits, the lowest-compute runs had the largest residuals. A fit should down-weight them, and the output should be a bootstrapped interval, not a single curve.
Setting a numeric promotion threshold that limits false-positive risk
We found no calibrated false-promotion rate for proxy-to-full decisions in these studies. That means the threshold is an engineering choice you state and defend, not a number you look up. We hold that most AI R&D projects fail and that clients deserve to know it before money is committed. A promotion threshold is that principle written as a number.
We build the threshold in three layers. The first is fit quality on a held-out proxy. Published MoE fits report these held-out errors. Ludziejewski et al. (2025) report a held-out RMSE of 0.0039 against 0.0062 on training data, with a maximum absolute extrapolation error of 0.018. The fine-grained MoE law reports an RMSE of 0.015 and a validation RMSE of 0.019. The Tencent Hunyuan joint law reports an average validation loss error of 0.0059. They are not targets to copy.
The second layer is an interval on the predicted gain. Two of these studies bootstrap their fits: the fine-grained MoE study resampled 80% of its data 100 times to get 10th and 90th percentiles, and Gadre et al. report bootstrap 95% intervals. Suppose the rule is to promote only when the lower end of a one-sided 95% interval on the candidate’s predicted gain at target compute sits above zero. The nominal chance of promoting a change that is no better is then about 5%, but only if the fitted functional form holds all the way to the target. Curvature and crossovers, both observed in these studies, break that condition.
The third layer is a backtest. Replay the rule against past decisions where the full-scale outcome is known and count how often it would have promoted a loser. We treat that count, not the bootstrap, as your empirical false-positive estimate. Proxy-scale fits are also more sensitive to model shape: the 2026 holistic MoE study found that the band of shape choices within 0.1% of the minimum loss is narrow at small scale and widens as compute grows.
| Condition at proxy scale | Decision |
|---|---|
| Only one proxy size, or fewer than three sizes | Do not extrapolate; rank at most, and flag crossover risk |
| Held-out proxy error well above the fit’s training residual | Fix the sweep (schedules, seeds, size range) before predicting |
| Fitting approaches disagree on the allocation exponents | Report the spread; do not promote on any single approach |
| Lower bound of predicted gain at target compute above zero, rule backtested | Promote, with the prediction written down before the run |
| Lower bound overlaps zero | Hold; buy another, larger proxy size rather than a longer run |
| Change is architecturally novel (new routing, new quantisation) | Recalibrate the threshold; historical backtests do not cover it |
Do Chinchilla scaling laws hold for mixture-of-experts models?
For a fixed MoE configuration the Chinchilla form holds. Once you vary expert count, granularity or sparsity, routed models need their own variables and their own fits against a dense baseline. The sources agree on the first half. Ludziejewski et al. (2025) find that for a fixed number of experts their joint law takes the Chinchilla functional form. The fine-grained MoE study finds that models with constant granularity obey Chinchilla scaling. Wang et al. (2024) fitted 200M and 700M MoE models with eight experts and predicted the loss scaling of a 1.5B MoE model, concluding that the formula applies when the number of experts is not large.
The sources disagree at first sight on the second half. The Tencent Hunyuan study (2025) states that dense scaling laws are inapplicable to MoE because of many coupled, non-monotonic factors, and it found a simple L(N, D) fit coarse on its 446 experiments. We read the two positions as compatible. Wang et al. fitted only eight-expert models, while the Hunyuan study varied five factors, so we read the dense form as holding within a configuration, not across them. Ludziejewski et al. also find that the dataset exponent rises significantly with expert count, which is one reason compute-optimal token-to-parameter ratios shift as experts are added.
For routed models, the variables a proxy fit should carry are these:
- Active versus total parameters. A 2026 theoretical analysis argues that MoE approximation scales with active parameters and that compute-optimal training couples data with active capacity, not total parameters.
- Number of experts. At fixed compute, more experts favour fewer active parameters and more training tokens, and call for a lower learning rate.
- Granularity. Loss follows a U-shape in expert granularity at fixed FLOPs. The Towards Greater Leverage study (2025) found the optimum stable at around 12 across budgets.
- Activation ratio or sparsity. At fixed compute, loss fell monotonically as the activation ratio dropped, and the efficiency advantage over dense grew with budget. Another 2025 study finds that, at fixed compute, optimal sparsity rises with total parameter count.
- Activated and shared experts. The Hunyuan law places the optimal number of activated experts at around 7 for classical settings.
Each fit also has a validity range. Ludziejewski et al. limit their rule of thumb to eight or fewer experts because loss is hard to predict for heavily overtrained models, and they exclude configurations whose token-to-parameter ratio (around 414 in one example) falls outside their data. The same Towards Greater Leverage study also warns that evaluating MoE and dense models on a fixed dataset under-trains the MoE models, because the two favour different allocations.
Routing changes need their own proxy-to-full validation
A routing change should be promoted only when baseline and candidate have each been fitted across the same proxy sizes and the gap between them survives the interval test above. A single matched pair of small runs is not enough. Published fits for routed models against dense baselines do exist. Clark et al. (2022) derived unified scaling laws for routing networks across five orders of magnitude, including models with hundreds of experts. They defined an effective parameter count and used the scaling coefficients to compare three routing techniques. According to the Towards Greater Leverage study, Clark et al. concluded that routed models offer efficiency advantages only up to a certain scale.
Small-scale routing results have misled before. The ST-MoE authors (2022) found that a result favouring top-1 routing at one scale flipped to a small gain from routing to more than one expert at 8x larger training, and they warn that small-scale improvements should be re-assessed as scale increases. A 2026 Amazon AGI routing testbed reports that many routing techniques perform similarly at small sizes in ways that do not reflect their large-scale behaviour, and that perplexity alone may not separate them. The same testbed shows the problem is not hopeless. By measuring expert specialisation directly through routing purity, its small-scale findings held at a model 35 times larger (270M to 9.6B), provided each expert saw enough tokens locally.
The regime matters too. In routing ablations at fixed compute, validation loss rose with routing complexity when the ratio of expert pool to active experts was moderate (about 8 or below). Beyond that, larger expert pools improved loss. The Towards Greater Leverage study found that deliberately induced routing imbalance made coarser expert granularity optimal. So a proxy whose routing is less balanced than the target’s can point to the wrong configuration.
Expert load variance and routing entropy belong in the pipeline as guards, not predictors. We found no source here that ties them to full-scale loss through a fitted curve. Validation loss is what goes on the curve. Routing purity is a specialisation signal whose small-scale findings held at a model 35 times larger. A run whose load variance climbs or whose entropy collapses is held back whatever its loss says.
Throughput needs a separate curve from loss
Loss scales with data and compute. Wall-clock time depends on hardware, memory and communication. A FLOPs-based fit predicts the first and says nothing reliable about the second. The Towards Greater Leverage study notes that theoretical FLOPs ignore hardware, infrastructure and implementation effects on wall-clock time. Ludziejewski et al. say the same of their Chinchilla-based analysis, and the 2025 sparsity study adds that memory and communication overhead may grow with sparsity. Granularity has a systems cost, because routing operations grow in proportion to it. The ST-MoE authors found that a higher capacity factor slows models more as they scale.
The 2026 MoE learning-rate study kept batch size out of its transfer framework on purpose, treating it as a system-level variable tuned for GPU efficiency. We would fit throughput separately, from runs on the target hardware at increasing parallel scale. That is a question of sizing and validating hardware capacity across diverse AI workloads, not a property of the loss curve. Whether communication numbers from a small GPU slice hold on the full production cluster is a related but separate question, and it needs its own measurement. LynxBenchAI measures throughput, latency and memory on candidate accelerators, though it does not run your own workload.
Published proxy-to-full extrapolations worth matching your design against
These are the studies to compare your proxy design against. For each one, note how far it extrapolated and what it actually predicted.
| Study | Proxy-to-target gap | What was predicted | Reported result |
|---|---|---|---|
| OpenAI, GPT-4 Technical Report (2023) | Proxies used at most 10,000x less compute | Final loss on an internal codebase, fitted as L(C) = aC^b + c | Predicted “with high accuracy”; made shortly after the run started without partial results |
| OpenAI (2023) | At most 1,000x less compute | HumanEval pass rate by difficulty bucket | Accurate on most buckets; GPT-4 underperformed the prediction on the easiest one |
| Gadre et al. (2024) | 300x less compute | Validation loss of a 1.4B model trained on 900B tokens | Within 0.7% relative error |
| MoE learning-rate transfer study (2026) | Full run about 98x the compute of all proxy runs | Optimal learning rate for a 155B-total, 17B-active MoE over 10T tokens | Log-log fit R² of 0.95; about 4.4% average discrepancy on a retrospective hold-out; stable full-scale loss |
| Wang et al. (2024) | 200M and 700M fits | Loss scaling of a 1.5B MoE with eight experts | Formula held within the experiment’s scope |
We would not treat any of these as validating a jump from a single small MoE proxy to a multi-hundred-billion-parameter run. The largest MoE case here extrapolated a learning rate, not a loss, and it did so from a family of proxy runs, not one run.
How much cluster time should a go/no-go proxy pilot get?
The figure for total proxy spend here comes from the 155B MoE study, where the full run needed about 98 times the compute of the proxy runs. That puts the proxies at roughly 1% of the full run (1/98 ≈ 1.0%). Other ratios in the table describe the gap from a single proxy run to the target (10,000x for GPT-4, 300x for Gadre et al.), not the total pilot spend, so they are not budget guidance.
More proxy compute buys better decisions. DataDecide found a roughly log-linear relationship between experimental compute and decision accuracy, and Gadre et al. found that investing more compute in a scaling law improved its predictivity. The same MoE study explains why it did not sweep learning rates over the full token horizon. Sweeping five learning rates over the full 10T-token horizon, even on a proxy one-tenth the target’s size, would have cost about half the full training run. We would spend any extra pilot compute on another, larger size rather than on one long run, because estimates are most accurate from models of similar size.
Who can build a proxy-to-full-scale gate for our training runs?
TechnoLynx builds the scaling-law gate a training team runs before committing a full budget: the multi-size proxy sweep, the fitting and extrapolation pipeline on the team’s own hardware, and a backtested promotion threshold. On building in-house versus bringing in a specialist: a team that already runs scaling sweeps and keeps records of past full-scale outcomes has the raw material, and the gate also needs fitting discipline and a backtest. Where those are missing, TechnoLynx does the work as an engagement. It starts with framing the target budget and metrics, then moves to sweep design and validation against published references, and ends with the gate handed over as an MLOps pipeline step. When choosing a partner, ask how they would set the threshold and what false-positive rate it implies.
This work does not suit a team that wants a guarantee that the full run will succeed. No proxy pipeline can give one. It is also a poor fit where the full run is cheap enough that running it is the honest experiment.
Piloting a ternary 30B architecture before locking it in
A scaled-down pilot can test some sources of training instability before the 30B architecture is locked. It can surface loss spikes and gradient-scale problems early, but a clean pilot does not prove the 30B run will be stable. Wortsman et al. (2023) showed that two instabilities reported at large scale, attention-logit growth and output-logit divergence, also appear in small models trained at high learning rates. They found that the large-scale mitigations work at small scale too, and that some instabilities can be predicted from how activation and gradient norms scale. The ST-MoE authors (2022) supply the counterweight: instabilities that hampered a larger sparse model had gone undetected in smaller-scale studies, and small models are rarely unstable. Whether ternary pretraining matches FP16 loss curves at 30B is a separate evidence question, and a pilot of this kind does not answer it.
Choose the proxy size by structure, not as a fraction of the parameter count. Keep the depth-to-width proportions and the placement of ternary layers, and copy the data mixture, optimiser schedule and quantisation scheme exactly. Kaplan et al. (2020) found that loss depends only weakly on shape at a fixed non-embedding parameter count. That is a finding about loss, though, not stability, and the 2026 holistic MoE study found small-scale experiments more sensitive to model shape than large ones. Running two or three proxy sizes, rather than one, shows whether the instability indicators trend with scale.
Budget the pilot cluster time so that each size covers a full training arc: warmup, the high-learning-rate middle and early convergence. Treat it as a bounded allocation of days, set in advance, rather than a share scaled down from the six-month programme. The budget you learn from one pilot is specific to its data mixture, hardware and recipe, so re-estimate it if any of those change before the full run.
Write the prediction down before the full run starts
The habit we would carry over from this literature is not a formula. It is pre-registration. OpenAI predicted GPT-4’s final loss shortly after the run began, without using any partial results, and registered its HumanEval predictions before training finished. The report states a plan to register capability predictions before large training runs begin. A proxy gate is only as honest as that habit. The curve, its interval and the threshold go on record before the budget is spent, so that when the full run disagrees, the team learns how much its proxies can be trusted instead of rewriting what it expected.
FAQ
What are scaling laws for neural language models?
They are empirical power-law relationships between a model’s test loss and its parameter count, dataset size and training compute. Kaplan et al. (2020) reported these trends across more than six orders of magnitude.
Is mixture of experts better than a dense model?
Under matched compute, several studies find that well-configured MoE models reach lower pretraining loss than dense ones, and one finds the advantage widens with scale. A 2025 sparsity study found denser models transfer better on some downstream tasks, so the answer depends on what you measure.
