Build a receipt scanner and bill splitter

Six people, one crumpled receipt, and somebody is about to open a calculator app. Every bill splitting product starts at that table, and the engineering underneath it is smaller than it looks: one extraction job, then arithmetic you should absolutely write yourself.

This is the second post in a short series on consumer apps that are mostly one API call. The first built a meal photo calorie tracker.

The pipeline

1. One job. Photo of the receipt, schema back with line items, tax, tip, total and the fiscal QR payload where one is printed.
2. Your code. Assign items to people, then split the shared parts to the cent.

There is no second job here, and it would be dishonest to invent one. Once you have typed line items, the rest is a UI and a rounding policy.

The job

{
  "sources": [{ "type": "image", "data": "data:image/jpeg;base64,/9j/4AAQ..." }],
  "engine": "vera-1.0-high",
  "schema": {
    "type": "object",
    "properties": {
      "merchant":  { "type": "string" },
      "paid_at":   { "type": "string", "format": "date-time" },
      "currency":  { "type": "string", "format": "currency" },
      "line_items": {
        "type": "array",
        "description": "One entry per printed line. Do not merge lines.",
        "items": {
          "type": "object",
          "properties": {
            "description": { "type": "string" },
            "quantity":    { "type": "integer", "minimum": 1 },
            "unit_price":  { "type": "number", "multipleOf": 0.01 },
            "amount":      { "type": "number", "multipleOf": 0.01 }
          },
          "required": ["description", "amount"]
        }
      },
      "subtotal":  { "type": "number", "multipleOf": 0.01 },
      "tax":       { "type": "number", "multipleOf": 0.01 },
      "tip":       { "type": "number", "multipleOf": 0.01 },
      "total":     { "type": "number", "multipleOf": 0.01 },
      "fiscal_qr": { "type": "string", "format": "qrcode" }
    },
    "required": ["total"]
  }
}

multipleOf: 0.01 is not decoration. Money that goes through a float arrives as 110.60000000000001 unless something pins it, and the runtime snaps every numeric field to its declared step after extraction. A line amount printed as quantity times unit price is computed in code as well, rather than worked out by a model in its head.

format: "currency" maps a euro sign or the word "euros" to EUR. A till timestamp in any layout becomes ISO 8601 through format: "date-time". Neither guesses: a value that does not resolve comes back exactly as printed, which is visible to you and can be routed to review.

fiscal_qr is the interesting one. Portugal, Spain, Italy, India and a growing list of countries now mandate a QR code on the receipt carrying its tax data, and that payload is decoded by a real barcode decoder rather than read off the pixels by a model. If the receipt has no code, the field is null rather than something that looks like a payload.

Which gives you a free correctness check on the field that matters most:

//portuguese fiscal qr: star separated fields, O is the total with taxes
function totalFromFiscalQr(payload: string): number | null {
  const field = payload.split("*").find((part) => part.startsWith("O:"))
  return field ? Number(field.slice(2)) : null
}

const printed = receipt.total
const declared = receipt.fiscal_qr ? totalFromFiscalQr(receipt.fiscal_qr) : null
if (declared !== null && Math.abs(declared - printed) > 0.005) {
  await flagForReview(receipt)
}

Where there is no fiscal code, the receipt still checks itself. Line items should sum to the subtotal, and subtotal plus tax plus tip should be the total. Two comparisons, no model, and they catch the reading mistakes that correct arithmetic cannot.

The split, in your code

Here is the part every tutorial skips. Once your users have tapped which items are theirs, you have shares of the subtotal, and you still have to distribute tax and tip proportionally and land on integer cents that sum exactly to the total.

Naive rounding loses money. Three people splitting a $10.00 charge get $3.33 each and the bill is a cent short. The fix is the largest remainder method, and it is short enough to paste:

/** split a cent amount by weights, exact to the cent */
function allocate(totalCents: number, weights: number[]): number[] {
  const weightSum = weights.reduce((sum, weight) => sum + weight, 0)
  if (weightSum === 0) return weights.map(() => 0)

  const exact = weights.map((weight) => (totalCents * weight) / weightSum)
  const shares = exact.map(Math.floor)
  let remainder = totalCents - shares.reduce((sum, share) => sum + share, 0)

  //hand the leftover cents to the largest fractional parts first
  const order = exact
    .map((value, index) => ({ index, fraction: value - Math.floor(value) }))
    .sort((a, b) => b.fraction - a.fraction)

  for (const entry of order) {
    if (remainder <= 0) break
    shares[entry.index] += 1
    remainder -= 1
  }
  return shares
}

Run it once over the item subtotals per person, then again over tax and tip using those subtotals as the weights. The result sums to the printed total every time, and it is a pure function you can unit test in a second. This belongs in your repository, not in any model, ours included.

What you still have to build

The assignment screen. Tapping items onto faces is the entire user experience of this category and no API gives it to you.

A tip and tax policy. Proportional to what each person ate is the common choice, but shared starters, a per person cover charge and someone who only had water all argue otherwise. Pick a rule, write it down, and make it visible in the UI, because the argument you are trying to end is about fairness rather than mathematics.

Group state and settle up. Who owes whom, across many bills, with payments in between. That is a ledger, and it is the actual product.

The failure paths. A job can end failed, and the hold is refunded in full when it does. A photo can be too dark to read. A field can come back null because it genuinely is not printed. Each of those needs a screen, and the honest one is usually a manual entry fallback.

Cost and shape

Jobs are asynchronous by design. POST /v1/jobs/create returns immediately with an id and status: "pending", then you either register a webhook_url or poll GET /v1/jobs/:jobId/get. For a phone app, the webhook into your backend plus a push notification is the better pattern, because the user has already put their phone back in their pocket.

Billing is per token rather than per page. The published figure we have is measured: $5.63 per 1,000 pages across the 1,651 page OmniDocBench run on vera-1.0-high, which is full page transcription and therefore the heaviest possible ask. A receipt against a schema this size produces far less output, and output is where the cost is.

Start here

The quickstart sends a first receipt in a few minutes. Schemas and formats lists every format, and QR codes and barcodes explains why the fiscal payload is decoded rather than read.

Next in this series: a barcode pantry tracker.