AI Calorie Trackers: How They Work, and Where They Go Wrong

What happens between your message and the number

Whether you send a photo, a voice note or a sentence, the same pipeline runs underneath. Only the entry point differs.

1. Recognition. A vision model reads the photograph and describes what is on the plate: the dish, its visible components, how they are prepared. If you sent a voice note it is transcribed first; if you sent text this step is skipped entirely, which is why text is the most reliable input.
2. Portion estimation. Each identified component gets an estimated amount, either from your own wording ("two eggs, a slice of rye") or from visual proportions against reference objects in the frame — plate rim, cutlery, hand.
3. Database matching. Every component is resolved to an entry in a nutrition database. Public reference data such as USDA FoodData Central and Open Food Facts covers most whole and packaged foods; branded and restaurant items usually need a web lookup against published menu data.
4. Arithmetic. Per-100g values from each matched entry are multiplied by the estimated amounts and totalled into calories, protein, carbohydrate and fat, then added to your daily figure.

Step one is the genuinely modern part and it is largely solved: identifying that a plate holds salmon, rice and roasted vegetables is something current vision models do reliably. Steps two and three are estimation and lookup — old problems, imperfectly solved, and the source of nearly all the error.

Where the error actually comes from

It is worth being blunt about the ranking, because it contradicts the intuition that a better camera or a smarter model would fix things.

Portion estimation is the largest single source of error, and it is a genuinely hard geometry problem: a photograph is a two-dimensional projection, and the depth of a bowl or the height of a rice mound is not recoverable from it with confidence. Some tools use the phone's depth sensor to attack this directly. Even then, a hundred grams of rice and a hundred and eighty grams of rice look far more similar than the eighty-calorie difference between them suggests.

Invisible ingredients come second, and they are worse because nothing in the image hints at them. Cooking fat is the classic case: the same vegetables, roasted dry or finished in three tablespoons of olive oil, are visually identical and differ by roughly three hundred calories. Sauces, dressings, and anything mixed into a stew or a curry have the same property. No model can see what is not there to be seen.

Third, and unavoidable for everyone, is the database itself. A nutrition database entry is a representative average, not a measurement of your food, and the labels feeding it carry legal tolerance: printed values may deviate from true values by up to 20 percent in the United States, with comparable tolerances in Europe. That floor applies to every calorie tool ever built, AI or not, and it is why we set out our own sources and their limits on the accuracy page.

The tell that separates a careful tool from a confident one

There is one behaviour that reveals more than any accuracy claim: what the tool does when the meal is genuinely ambiguous.

A creamy soup photographed from above is not estimable. Neither is a stir-fry, where the amount of oil is the dominant calorie term and is completely invisible. A tool that returns an instant, precise-looking number for those is not more capable than one that hesitates — it is making the same guess and declining to tell you. A tool that asks whether the sauce was cream-based, or how much oil went into the pan, is doing the correct thing, and the mild annoyance of the follow-up question is the price of an estimate worth having.

Test this deliberately. Send a photo of a mixed dish — a curry, a stew, a bowl of pasta with sauce — and see whether the tool asks anything. Then send the same dish described in words with the amounts you actually used, and compare. If the two answers are far apart and nothing was asked, the photo number was a guess wearing a decimal point.

What good practice looks like on your side

Most of the accuracy available to you is not in the choice of tool. It is in what you give it.

• Where you know an amount, say it. "120g chicken breast" beats any photograph of chicken breast, every time, in any tool.
• Name the cooking method and the fat. "Fried in butter" or "dry-roasted" changes the answer more than the ingredient list does.
• Photograph packaged food's label rather than the food. A label resolves to an exact database entry; the contents of a bowl resolve to an estimate.
• Shoot at an angle, not straight down, and get the whole plate in frame with something of known size beside it. Depth is what the model is missing.
• Correct the estimate when it is wrong. A tool you can edit converges on your real eating; one you cannot compounds its own errors into your weekly totals.

Correction matters more than it sounds. Over a week, a diary you have nudged toward reality four or five times is substantially better than one you accepted passively, regardless of which product produced the first draft.

What the numbers are actually for

The point of tracking is not to produce an accurate number for Tuesday's lunch. It is to see a pattern over weeks — that dinners run larger than you thought, that protein collapses at breakfast, that weekends undo the week. Those patterns survive a fair amount of per-meal error, because the errors are noisy rather than systematic and largely cancel out across dozens of meals.

That is also why consistency beats precision. A slightly rough estimate you actually record every day tells you far more than a meticulous weighed entry you keep for four days and abandon. If you want a target to read those weekly totals against, our calorie calculator runs the Mifflin-St Jeor equation in the browser with no account, and there is more on the practice itself in our guide to counting calories.