SnapCalorie taps AI to estimate the caloric content of food from photos

While working at Google, Wade Norris wanted to create a project that could positively impact people’s lives. He co-founded Google Lens, Google’s computer vision-powered app that brings up information related to the objects it identifies. But it didn’t quite scratch the itch.

So several years ago, Norris teamed up with Scott Baron, a systems engineer in the aerospace industry, to launch a health-focused startup called SnapCalorie. SnapCalorie, powered by AI, attempts to get an accurate calorie count and macronutrient breakdown of a meal from a single photo taken with a smartphone.

This month, SnapCalorie raised $2 million in funding from investors including Accel, Index Ventures, former CrossFit CEO Eric Roza and Y Combinator. The company previously raised $125,000 from unidentified investors in a pre-seed round.

“Human beings are terrible at visually estimating the portion size of a plate of food,” Norris said. “SnapCalorie improves on the status quo by combining a variety of new technologies and algorithms.”

To be clear, SnapCalorie isn’t the first computer vision-based app for calorie counting. Apps such as Calorie Mama, Lose It, Foodadviser and Bite.AI have all attempted the feat — to varying degrees of success. But what makes SnapCalorie different, Norris claims, is its use of depth sensors on supported devices for measuring portion size and team of human reviewers for “an added layer of quality.”

“On average, the team is able to reduce the caloric error to under 20%,” Norris says. “There are other apps capable of using AI to do photo-based meal tracking, but none of them help with portion size estimation — the most important part to reduce error.”

There’s a lot of skepticism in the health industry around photo-driven calorie estimating tools — and for good reason. One 2020 study comparing some of the more popular AI-based calorie counters found that the most accurate — Calorie Mama — was only right about 63% of the time.

So how’s SnapCalorie improved? Beyond the use of depth sensors and reviewers, Norris points to an algorithm that the company developed that can ostensibly outperform a person at estimating a food’s calories. Using the algorithm, SnapCalorie both identifies the types of food in a photo and measures the portion size of each to estimate the caloric content.

The results can be logged in SnapCalorie’s food journal or exported to fitness-tracking platforms like Apple Health .

The algorithm’s reported strong performance comes from its unique training data set of 5,000 meals, Norris says, which SnapCalorie created by taking thousands of photos of each meal — e.g. soups, burritos, oils, “mystery sauces” and more — using a robotic rig.

“We made sure these had all of the diverse and challenging conditions you’d see in the real world and we weighed out every single ingredient on a scale,” Norris said. “The traditional pipeline for training an AI model is to download public web images, have people label the images and then train the model to predict those labels. This isn’t possible for food, because people are very inaccurate at visually estimating portion size, so you can’t have people label the images after the fact.”

Norris admits that SnapCalorie’s algorithm may be biased toward American food, since the team collected most of the initial training data in the U.S. But the company in the process of expanding the training data — drawing both on photos from SnapCalorie’s users and internal data — to include other cultural cuisines, he says.

One might argue that, no matter how accurate the algorithm, no app can give a truly accurate account of how many calories you ate in a meal. There’s a range of variables apps don’t consider, after all, like different cooking methods and the amount of time it takes to break down individual foods.

Norris doesn’t make the claim that SnapCalorie is 100% accurate, suggesting that that app’s calorie estimating tools should be considered simply a piece of the larger nutrition puzzle. He spotlighted SnapCalorie’s other major feature, a ChatGPT-powered chatbot, which gives meal suggestions informed by a user’s coals and past preferences as well as SnapCalorie’s database of nutritional values.

“We’ve found that people’s interest in understanding what they’re putting in their bodies is on the rise. The negative health impacts of things like processed foods are becoming more and more clear every day,” Norris said. “We’ve heard that our users really like SnapCalorie especially when eating out, as many restaurants don’t post nutrition values, and they would otherwise have no way of logging the meal.”

To his point about popularity, SnapCalorie appears to be growing at a healthy clip — it’s on track to break 1,000 new users this month. The company’s focused on expansion at the moment as opposed to monetization, but Norris described the burn rate as “very conservative.”

“Our incredible organic growth rate seems to be indicative of our value proposition resonating well with consumers — people try it, love it, and recommend it to their friends and family,” he said.

SnapCalorie taps AI to estimate the caloric content of food from photos by Kyle Wiggers originally published on TechCrunch