Build a Cal AI clone: meal photo to calories and macros

A photo of a plate carries no serving size, no barcode and no nutrition label. Getting from those pixels to grams of protein means naming what is on the plate, estimating how much of it there is, looking each thing up somewhere authoritative, and doing the multiplication.

Three steps. Two of them are a Veralens job, and the one in the middle is a free public API.

The pipeline

1. Job one. Send the photo with a schema that returns the food items and an estimated quantity for each one.
2. Your code. Look each item up in a nutrition database. Open Food Facts is free and needs no key.
3. Job two. Send the first result plus the nutrition rows back with a final schema, and get calories and macros per item and the totals.

Nothing else runs on a server. The rest of the app is a camera button, a list and a chart.

Job one: what is on the plate

The photo goes in as a data URL. Raster only, so a JPEG straight off the camera is exactly right.

curl -X POST https://api.veralens.ai/v1/jobs/create \
  -H "Authorization: Bearer vl_your_secret_key" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": [{ "type": "image", "data": "data:image/jpeg;base64,/9j/4AAQ..." }],
    "engine": "vera-1.0-high",
    "schema": {
      "type": "object",
      "properties": {
        "items": {
          "type": "array",
          "description": "One entry per distinct food on the plate.",
          "items": {
            "type": "object",
            "properties": {
              "label":        { "type": "string", "description": "Short name as a person would say it, e.g. grilled chicken breast." },
              "search_terms": { "type": "string", "description": "Two or three words for a food database search. No brand unless the packaging is visible." },
              "preparation":  { "type": "string", "enum": ["raw", "boiled", "steamed", "grilled", "fried", "baked", "unknown"] },
              "grams":        { "type": "number", "multipleOf": 1, "minimum": 0, "description": "Estimated edible weight in grams." },
              "certainty":    { "type": "string", "enum": ["high", "medium", "low"] }
            },
            "required": ["label", "search_terms", "grams"]
          }
        }
      },
      "required": ["items"]
    }
  }'

Four things in that schema are doing real work.

search_terms exists because the string a person wants to read and the string a database wants to match are not the same string. "Grandma's roast potatoes" is a good label and a terrible query.

preparation is an enum because it changes the numbers a lot. Boiled and fried are not the same food, and a closed set gives you something to branch on instead of free text you have to normalize.

grams carries multipleOf: 1 and minimum: 0, which the runtime applies after extraction. You get whole grams, never a negative one, and never 180.00000000000003.

certainty is the field that decides your UI. Anything marked low gets shown to the user for confirmation before it counts.

The job comes back immediately with an id and status: "pending". Register a webhook_url if your backend can receive one, and poll GET /v1/jobs/:jobId/get if it cannot.

{
  "items": [
    { "label": "grilled chicken breast", "search_terms": "chicken breast grilled", "preparation": "grilled", "grams": 180, "certainty": "high" },
    { "label": "white rice",             "search_terms": "white rice cooked",      "preparation": "boiled",  "grams": 150, "certainty": "medium" },
    { "label": "steamed broccoli",       "search_terms": "broccoli steamed",       "preparation": "steamed", "grams": 90,  "certainty": "high" }
  ]
}

The middle: look the food up

Open Food Facts has a public read API. No key, no signup. A text search looks like this:

async function findNutrition(searchTerms: string) {
  const url = new URL("https://world.openfoodfacts.org/cgi/search.pl")
  url.searchParams.set("search_terms", searchTerms)
  url.searchParams.set("json", "1")
  url.searchParams.set("page_size", "5")
  url.searchParams.set("fields", "product_name,nutriments,nutrition_data_per")

  const response = await fetch(url, {
    //open food facts asks every client to identify itself
    headers: { "User-Agent": "MealApp/1.0 (hello@yourapp.com)" },
  })
  const body = await response.json()
  return body.products ?? []
}

What comes back, for the fields you care about, is per 100 grams: energy-kcal_100g, proteins_100g, carbohydrates_100g, fat_100g.

Two honest caveats about this database, because they will shape your product.

Open Food Facts is built around packaged goods. Scan a cereal box and it is excellent. Search it for "grilled chicken breast" and you are relying on whatever community entries exist, which is thinner and noisier. For raw and cooked ingredients, USDA FoodData Central is the better source: it needs a free API key from api.data.gov, and its Foundation Foods and SR Legacy tables are curated rather than crowdsourced. Plenty of apps query both and prefer whichever returns a confident match.

The second caveat is that matching is yours. Deciding that "chicken breast grilled" means row 4 and not row 1 is a ranking problem in your code, and it is the part of this app that will take you the longest. Start with the first result that actually has the four nutriment fields populated, ship it, and improve it once you can see where it is wrong.

Job two: the numbers

Now hand both halves back as text and let the schema produce the finished object.

const BASE = "https://api.veralens.ai/v1"

const context = [
  "DETECTED ITEMS",
  JSON.stringify(detected.items),
  "",
  "NUTRITION ROWS (per 100 g)",
  JSON.stringify(nutritionRows),
].join("\n")

const response = await fetch(`${BASE}/jobs/create`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.VERALENS_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    sources: [{ type: "text", data: context }],
    engine: "vera-1.0-high",
    schema: mealSchema,
  }),
})
const { data: job } = await response.json()

And mealSchema:

{
  "type": "object",
  "properties": {
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "label":     { "type": "string" },
          "grams":     { "type": "number", "multipleOf": 1 },
          "matched":   { "type": "string", "description": "Which nutrition row was used. Null when no row fits." },
          "calories":  { "type": "integer", "description": "grams / 100 multiplied by the row kcal per 100 g." },
          "protein_g": { "type": "number", "multipleOf": 0.1 },
          "carbs_g":   { "type": "number", "multipleOf": 0.1 },
          "fat_g":     { "type": "number", "multipleOf": 0.1 }
        },
        "required": ["label", "grams"]
      }
    },
    "totals": {
      "type": "object",
      "properties": {
        "calories":  { "type": "integer", "description": "Sum of the item calories." },
        "protein_g": { "type": "number", "multipleOf": 0.1 },
        "carbs_g":   { "type": "number", "multipleOf": 0.1 },
        "fat_g":     { "type": "number", "multipleOf": 0.1 }
      }
    }
  },
  "required": ["items", "totals"]
}

{
  "items": [
    { "label": "grilled chicken breast", "grams": 180, "matched": "chicken breast grilled", "calories": 297, "protein_g": 55.8, "carbs_g": 0, "fat_g": 6.5 },
    { "label": "white rice", "grams": 150, "matched": "white rice cooked", "calories": 195, "protein_g": 4.1, "carbs_g": 42, "fat_g": 0.5 },
    { "label": "steamed broccoli", "grams": 90, "matched": "broccoli steamed", "calories": 32, "protein_g": 2.2, "carbs_g": 6.3, "fat_g": 0.4 }
  ],
  "totals": { "calories": 524, "protein_g": 62, "carbs_g": 48.3, "fat_g": 7.3 }
}

Why this step is a job and not a for loop

It could be a for loop. If your matching is confident you can multiply in your own code and skip job two entirely, and for a first version that is a perfectly good decision.

What the second job buys you is the messy middle. Five nutrition rows came back for "white rice" and one of them is rice pudding. One item found nothing at all. Another matched a row whose values are per portion rather than per 100 grams. Reading a list of candidate rows and picking the one that fits the food you saw is judgement, and judgement is what the engine is for. The matched field exists so that judgement is auditable rather than invisible.

The arithmetic is not judgement. When a field is derived, the model's job is to identify the quantities and how they combine, and the calculation itself runs in our runtime, in code: deterministic, correct precedence, rounded once at the end. That matters more here than it does on an invoice, because nothing in a nutrition total is printed anywhere for you to check it against. If the number is quietly wrong by four percent, you will never find out and neither will your user.

One rounding note. Per item macros rounded to one decimal will not always sum to the rounded total, because the total is computed from the unrounded values. If your screen shows both, recompute the total from the numbers you are displaying so the column adds up.

What you still have to build

The two jobs are maybe forty lines including the fetch wrapper. The app is not.

The grams control. Portion weight from a photo is an estimate and always will be. Every app in this category eventually ships a slider, and the good ones recompute on the client from the per 100 gram rows you already fetched, so adjusting a portion is instant and costs nothing.

The matching layer. Covered above. This is your moat, such as it is.

The error paths. A job can come back failed, and the balance hold is refunded in full when it does. Open Food Facts can return nothing usable. The photo can be a picture of a cat. Decide up front what each of those shows the user, because a spinner that never resolves is the default if you do not.

Everything a person sees. Camera, history, daily targets, streaks, the chart. That is the whole product as far as your user is concerned, and none of it is an API call.

Cost

Veralens bills per token, so a job costs what it consumes rather than a flat per image rate. The only figure we publish is measured: the full 1,651 page OmniDocBench run through vera-1.0-high came to $5.63 per 1,000 pages. That workload is whole page transcription, the most output heavy request there is, and a photo answered with a six field schema sits well under it because output is where the money goes.

Job two takes text in and produces a small object, so it is a sensible place to try vera-1.0-low and compare the two on your own meals.

Start here

The quickstart has job one running in a few minutes. Schemas and formats is the full list of what format and the numeric keywords do for you, and jobs and queue covers webhooks, retries and the failure path.

Next in this series: a receipt scanner and bill splitter.