The same open-weight model is sold by twenty providers at prices eight times apart, and two endpoints with the same model name can return different outputs for the same prompt. Bitdepth tests every endpoint every night against a published reference, grades each one from what it returned, and shows the lowest price at each grade.
A grade from A to D for every endpoint, from the output it returned against the reference, not the label it declared. All four columns print with the letter, and the noise floor prints beside them, so a difference smaller than the endpoint's own night-to-night variation is never called a difference.
For each model, the lowest listed price among the endpoints holding A, and holding B, with the date the price was read.
Every endpoint, every night. A grade that changes is a dated event, reported the next morning with the column that moved. You hear about it before your users do.
120 fixed prompts and 24 generated that night from a recorded seed, in five families (arithmetic, JSON, string puzzles, a planted code in about 2,500 words, exact repetition), go to every endpoint of a model, pinned to one provider, fallbacks off, temperature 0.
Each output is compared word for word against the plurality of endpoints declaring full precision. Each family also has its own answer key, so task accuracy is scored on its own. Each endpoint is compared with itself night to night, which gives the noise floor.
A to D on four columns: match, task accuracy, declared precision held, reliability. The lowest column wins and the reason prints with the letter. U until seven nights exist. Letters are provisional until the first calibration publishes, then final under that version of the method.
Example figures are filled in. Put in your own.
The grade is what makes the cheaper endpoint safe to route to for what the probes test; without it the saving is a bet.
Tell me your model, your provider, and your monthly volume. You get back the measured grades, the price comparison, and the limitations for the endpoints you could be routing to.
hello@bitdepth.devA grade measures defined behaviors against a stated reference: did the endpoint return what the full-precision endpoints returned, did it answer the probes correctly against their answer keys, did its declared precision stay the same, did it respond. It does not rank models, it does not say an endpoint is best, and measured behavior is not proof of the weights or precision an endpoint runs. Declared precision is recorded as a claim and held to consistency over time. Thresholds, the reference rule, the noise rule, a worked example and the limits are on the method page.
Your endpoint is already in the comparison. Your team sees its results first and can flag factual errors before publication; publication then follows the same rules for every endpoint, whatever the grade. The policy is on the method page. Write to hello@bitdepth.dev with the person who should receive it.
Reserved blocks at a fixed price with a nightly grade, and the graded price table, are on the Market page.