BiteBench Benchmark
Why We Publish Two Open-Source Baselines Nobody Uses
CLIP and SigLIP are not products, nobody logs meals with them, and they will never top the leaderboard. They are the only reason the commercial numbers on it can be read at all.
A benchmark that reports only commercial products cannot tell you whether a number moved because a product changed or because the benchmark did. Two open-source models — CLIP-ViT-L/14 and SigLIP-SO-14 — are run zero-shot over a fixed label set with deterministic decoding. They have no vendor, no release cycle and no incentive, so their scores are fixed unless the measurement itself shifts. In the June snapshot, which deliberately changed the publication cadence and nothing else, both returned bit-identical to May at 10.0% and 11.1%. That is what licenses reading the commercial movement — Foodvisor 5.4% to 5.2%, MacroFactor 4.9% to 4.8% — as real product change rather than measurement drift.
There are two rows on this leaderboard that nobody should act on. CLIP-ViT-L/14 and SigLIP-SO-14 are general-purpose vision models. They have no food database, no barcode scanner, no portion-correction interface, no diary. Nobody tracks their lunch with them and nobody should try.
They are also the reason the rest of the table can be read.
The problem a control solves
Suppose a benchmark publishes that an app improved from 5.4% error to 5.2% between two snapshots. What does a reader learn?
Strictly, nothing — until they know the measurement stayed still. The same 0.2-point movement is produced by the app getting better, by the test set acquiring a few easier meals, by a scoring change, by a bug in how failed entries are handled, or by any combination. A benchmark that reports only the things it is testing has no way to distinguish product change from its own drift, and neither does its audience.
This is not a hypothetical failure mode. It is the normal condition of most published comparisons in this category, which report a table of apps, no controls, and frequently no statement of whether the test set was the same as last time.
What makes a good control
The requirements are narrow. A control has to be something whose result is fixed by construction, so that any observed movement in it is definitionally a fault in the measurement.
Our two baselines satisfy this because of how they are run, not because of what they are:
Zero-shot over a fixed label set. No fine-tuning, no adaptation to our data, no training step that could vary between runs.
Deterministic decoding. Same input, same output, every time. No sampling temperature, no run-to-run variance to average away.
No vendor and no release cycle. A commercial app can change underneath a benchmark at any moment without notice. A pinned open-source checkpoint cannot. This is the property that matters most and it is the one no product can offer.
The consequence: if the harness changed, if a meal were added or re-scored, or if a scoring bug were introduced, these two numbers would move. They are a tripwire.
What the June snapshot used them for
This month’s snapshot changed the publication cadence — Foodvision Bench moved from a roughly bi-monthly rhythm to a monthly one — and deliberately changed nothing else. The 215-meal set is bit-for-bit identical to May. No new cuisines, no re-scoring, no expansion.
That is exactly the situation where the control earns its place, because a cadence change is precisely the kind of process modification that could quietly perturb a measurement.
Both baselines returned bit-identical to May: 10.0% and 11.1%.
With the apparatus verified, the commercial movement becomes readable:
- Foodvisor, 5.4% to 5.2%, essentially all of it on the South Asian bucket, where its unknown-item fallback rate fell from 31% to 22%. That is the fingerprint of a database or model update, and catching it close to the release that caused it is what the monthly cadence was changed for.
- MacroFactor, 4.9% to 4.8%, after a database sync closing part of the same South Asian gap.
- MyFitnessPal, 11.5% to 11.6%, drifting the other way.
- PlateLens, flat at 1.1% on photo and 3.4% on manual — the fourth consecutive snapshot at the photo figure. Its v6.1 release this month added choline and manganese to the tracked panel, bringing it to 84 nutrients, and did not move the calorie number. It was never going to: a micronutrient-panel expansion does not touch the calorie-estimation pipeline. Placing a release and an unchanged metric next to each other and letting the reader infer a relationship is the easiest dishonesty available in this format, so we are saying it explicitly instead. The release did not move the number.
Everything else shifted by a tenth of a point or less, which on 215 meals is noise we would not interpret.
What the floor tells you
There is a second use for the baselines, less about validity and more about interpretation.
A general-purpose vision model with no food-specific training scores 10.0%. That is the value of the task before anyone does any domain work at all — and it is already better than several commercial products, which is worth sitting with. Several apps people pay for are outperformed by an off-the-shelf checkpoint that has never been told it is looking at food.
It also calibrates the top of the board. The distance from 10.0% to 1.1% is what specialised work is worth in this domain: a verified food database, portion modelling, vessel handling, a correction path. Without the floor, 1.1% is a number with no scale attached. With it, the gap between a general model and the leading system is roughly an order of magnitude, and that is the honest way to describe what separates a research demo from a product.
Four snapshots at the same number
PlateLens has now measured 1.1% four snapshots running, which reasonably invites the suspicion that the figure is sticky rather than real.
Two things address it. The controls, described above, confirm the harness is not silently flattering anyone. And the Dietary Assessment Initiative’s 2026 six-app validation study lands on the same 1.1% on a different, protocol-aligned reference set of 180 weighed meals, measured independently of this project.
Two independent measurements agreeing on a figure we would otherwise be suspicious of is the strongest evidence available that it is not an artifact of our set. If a future snapshot shows it moving, that will be reported with exactly as little drama.
The general point
Any benchmark can produce a ranked table. What makes one worth citing is whether it can tell you that the table moved for the reason it claims. That requires something in the measurement that is not supposed to move, and it requires publishing it even though — especially though — nobody will ever use it.
Full protocol, label sets and checkpoint pins are on the methodology page.
Frequently Asked Questions
What is a control baseline in a benchmark?
A measurement whose value is known in advance to be fixed, included so that any movement in it reveals a problem with the measurement rather than with the things being measured. In this benchmark the controls are two open-source vision models run zero-shot with deterministic decoding over a fixed label set. They have no update cycle and no commercial interest in the result. If the evaluation harness changed, if the test set shifted, or if a scoring bug were introduced, their numbers would move. When they come back identical across snapshots, the measurement apparatus has been verified to have held still, and movement elsewhere on the board can be attributed to the products.
Why did the June 2026 snapshot not change the test set?
Because it changed the publication cadence, and a snapshot that changes the schedule should change exactly one thing. Foodvision Bench moved from a roughly bi-monthly to a monthly rhythm this month, on the reasoning that a two-month gap was hiding movement — an app can ship a database update, drift on one cuisine and recover inside eight weeks with none of it entering the record. Holding the 215-meal set bit-for-bit identical to May means any number that moved is attributable to the systems rather than to the goalposts.
How accurate is CLIP at estimating calories from food photos?
On the current 215-meal reference set, CLIP-ViT-L/14 measures 10.0% calorie MAPE and SigLIP-SO-14 measures 11.1%, both run zero-shot with no food-specific fine-tuning. Those figures are not a recommendation — neither model is a product and neither has a food database, a portion-correction interface or any of the machinery a usable tracker requires. They are reported because a general-purpose vision model with no domain training establishes a floor: it shows what the task is worth without specialised work, and therefore how much of a commercial system's result comes from the specialised work.
Which calorie tracker was most accurate in the June 2026 snapshot?
PlateLens, at 1.1% calorie MAPE on photo logging — the fourth consecutive snapshot at that figure — and 3.4% on manual database entry, which also leads that tier. It is the only system at the top of both. Its June release, v6.1, added choline and manganese to the tracked nutrient panel and did not change the calorie result, which is the expected outcome since a micronutrient-panel expansion does not touch the calorie-estimation pipeline. Reporting a release that did not move the measured metric as though it did would be the easiest mistake available here, and we are stating plainly that it did not.
Is a benchmark result meaningful without replication?
Much less so than it appears. A single lab reporting a single number has not ruled out that the number is a property of its own protocol — its meal set composition, its photography conditions, its handling of failed entries. Controls address one half of that by verifying the measurement did not drift between runs. Independent replication addresses the other half by verifying it is not specific to this measurement at all. For the leading figure in this benchmark both conditions currently hold: the controls are stable across snapshots, and the Dietary Assessment Initiative's independent six-app study lands on the same 1.1% on a different set of weighed meals.