This page does not change to sell anything. Every grade on any Bitdepth surface is produced by the procedure below, and the procedure is versioned; the version prints with every grade.
A unit is a model at a provider-declared precision with a Bitdepth grade, plus the context and latency class it was observed serving. The declared precision and the grade are different attributes: one is a claim held to consistency, the other is a measurement. Anything written on a unit says which of the two it depends on.
One grade per model, endpoint and window (7 nights by default). M: match on the model's stable families. T: task accuracy, as points below the best endpoint for that model in the window. D: declared precision present and unchanged. R: reliability. Latency class and price are shown, not graded. All four column values print with the letter.
| Grade | M: match against reference | T: task accuracy | D: declared precision | R: reliability |
|---|---|---|---|---|
| A | Within the noise floor every night of the window | Within 2 points of the best endpoint | Present and unchanged | Errors under 1%, truncation under 2%, misrouted 0 |
| B | Within the noise floor every night | Within 5 points | Present or absent, unchanged | Errors under 3% |
| C | Outside the floor on 1 to 3 nights of 7 | 5 to 10 points below | Changed during the window, dated | Errors 3% to under 10% |
| D | Outside the floor on 4 or more nights of 7 | More than 10 points below | Errors at or above 10% | |
| U | Unrated: fewer than 7 nights, endpoint absent on 3 or more nights, or the model had no reference on more than 20% of prompts | |||
Separately for each model and each probe family, every endpoint's night-to-night disagreement with itself on the fixed prompts is recorded over the window. The noise floor for that model and family is the 75th percentile of those disagreements across endpoints. An endpoint's match is within noise when it is at or above 100 minus that floor.
The 75th percentile is used rather than the maximum so that one or two highly variable endpoints cannot widen the tolerance for everyone else; an endpoint whose own variation sits above the floor is, by construction, outside noise and reads C or D on M. A family is stable for a model when the median endpoint agrees with itself at or above the stability threshold; M is computed on stable families only, so a family that is noisy for everyone (on night one, arithmetic on one model) does not drag the match column for a model it cannot separate.
A difference smaller than the floor is not reported as a difference, anywhere.
Numbers are invented to show the arithmetic. Model X, seven nights. Night-to-night self-disagreement on the arithmetic family across its eleven endpoints: 0, 0, 1, 1, 2, 2, 3, 4, 6, 8 and 11 percent. The 75th percentile is 4, so the floor is 4 and within-noise means a match of 96 or better.
Declared precision is what the provider publishes on the router. It is recorded every night and treated as a claim: a grade holds it to consistency over time, not to truth. Measured behavior is what the endpoint returned. Agreement with the plurality is evidence that an endpoint served the same thing the reference endpoints served; it is not proof of the weights, quantization or serving configuration behind it, and Bitdepth does not claim to have verified those.
A grade is not a ranking of models and does not say which endpoint is best. Agreement with the plurality is not the same as correctness, which is why task accuracy is a separate column with its own answer keys. The probe families are narrow by design so that a difference is attributable; they do not cover reasoning depth, coding, tool use or long-task instruction following, and a grade is not a claim about them. The 24 prompts generated each night from a recorded seed limit tuning to the set.
A grade reads U until seven nights exist for the endpoint. Letters are provisional until the first calibration publishes the thresholds from the first fourteen nights; from then they are final under that version of the method, and the method version prints with every grade. A later version re-grades the archive and says so.
Numbers against a stated reference. Not a claim about intent. Nothing on this site describes why any provider serves the way it does.