How to parse unstructured data properly

Unstructured data is anything that does not arrive in rows and columns: a scanned invoice, a PDF contract, a photo of a receipt, an email body, a spreadsheet nobody agreed on a format for. Parsing it properly means ending up with typed values you can insert into a database without a human checking each one.

Below is what that costs you built by hand, step by step, and the same result in one request. Both with real numbers.

The manual way, step by step

Step 1: classify the document. You cannot use one schema for invoices, receipts and ID cards, so you need to know what arrived before you parse it. That is usually a second model call, or a filename convention that breaks the first time a customer uploads something unexpected. Costs you: an extra model call per document, plus the routing code.

Step 2: get the content out, and branch on the input. A digital PDF has a text layer you can read with a library. A scan does not, and needs OCR. A DOCX needs a different parser. A spreadsheet needs another. Costs you: three or four libraries, and a bug the week a photographed receipt arrives in a folder of generated PDFs.

Step 3: extract the fields. Prompt with your schema, hope the field names stay stable, and discover they do not. To get quality close to a purpose-built extractor you also have to restate your rules in every call: date conventions, decimal separators, when to abstain, how to treat enums. Costs you: input tokens on every document for rules that never change, plus the drift.

Step 4: normalize and validate. Dates to ISO with the ambiguity resolved from context, currencies and countries to ISO codes, phone numbers to E.164, IBAN checksums, rounding money without floating point artifacts. Costs you: a few hundred lines you own forever, and the per country cases you have not met yet.

Step 5: make it durable. Extraction takes seconds per page with a long tail, so a request cannot wait for it. You need a queue, retries with backoff that do not double-charge you, webhook delivery, and per document cost tracking. Costs you: infrastructure, and the incident where a retry storm bills you twice for work that already succeeded.

What step 3 actually costs

Step 3 is the one people underestimate, so we measured it. Same image, same fields, two requests: one carrying the rules a developer has to write to approach a purpose-built extractor's output, one from our API.

(Measured July 2026 on one page. The rule block in the first request is about 840 tokens.)

Those rules ride every single call. They do not change between documents, they are not about your document, and you pay for them a thousand times per thousand documents. That is roughly 80% more input than the same job costs here, before anything else in this post.

And that rule block is not even parity. It covers date conventions, decimal separators, ISO codes, abstention and enum handling. It does not cover the next section, because the next section cannot be written as a rule at all.

What no prompt can buy

Some of the gap is not a matter of writing better instructions. It is a matter of what a language model is:

Arithmetic. You can ask a model to compute a line total and verify it against the subtotal, and it will comply, and it will be wrong in the last decimal often enough to matter. There is no instruction that turns a text predictor into a calculator. Getting this right means computing in code, on the values that were read, which is a step in your pipeline and not a sentence in your prompt.

Reading a QR code or a barcode. A QR code is a grid encoding bytes through masking, interleaving and Reed-Solomon error correction. A model looking at it is doing pattern completion on something that is not text, and it will hand you a payload of the right shape with the wrong contents. No prompt fixes that. You need a decoder, which is another dependency, another step, and another thing that has to run before or after the model.

Checksum validation. An IBAN either passes mod-97 or it does not, and knowing which is arithmetic, not judgement. A model told to "verify the checksum" will tell you it verified the checksum.

Consistent abstention. Getting a model to return null rather than something plausible is the hardest behaviour to hold steady, and it is the one that decides whether you can trust the output without a human reading every document. It is not a line you add, it is something you tune and re-tune, and you find out you got it wrong from your data rather than from an error.

The first three are ours in code, deterministically, on every job. The fourth is what our extraction is built and continuously measured against. None of it is on your prompt bill, and none of it is in your repository.

The same thing in one call

You do not need to know what the document is before you send it. Describe every shape you accept in one schema, let each entry carry its own type and its own fields, and send whatever arrives:

{
  "type": "object",
  "properties": {
    "documents": {
      "type": "array",
      "description": "One entry per document in the batch.",
      "items": {
        "anyOf": [
          {
            "type": "object",
            "title": "invoice",
            "properties": {
              "kind":        { "type": "string", "enum": ["invoice"] },
              "supplier":    { "type": "string" },
              "invoice_number": { "type": "string" },
              "issue_date":  { "type": "string", "format": "date" },
              "currency":    { "type": "string", "format": "currency" },
              "total":       { "type": "number", "multipleOf": 0.01 }
            },
            "required": ["kind", "total"]
          },
          {
            "type": "object",
            "title": "receipt",
            "properties": {
              "kind":        { "type": "string", "enum": ["receipt"] },
              "merchant":    { "type": "string" },
              "paid_at":     { "type": "string", "format": "date-time" },
              "payment_method": { "type": "string", "enum": ["cash", "card", "transfer"] },
              "total":       { "type": "number", "multipleOf": 0.01 }
            },
            "required": ["kind", "total"]
          },
          {
            "type": "object",
            "title": "id_card",
            "properties": {
              "kind":         { "type": "string", "enum": ["id_card"] },
              "full_name":    { "type": "string" },
              "document_number": { "type": "string" },
              "nationality":  { "type": "string", "format": "country" },
              "date_of_birth":{ "type": "string", "format": "date" },
              "expires_at":   { "type": "string", "format": "date" }
            },
            "required": ["kind", "document_number"]
          }
        ]
      }
    }
  },
  "required": ["documents"]
}

Send a mixed batch to that schema and each entry comes back as the shape it actually is:

{
  "documents": [
    { "kind": "invoice", "supplier": "Acme GmbH", "invoice_number": "2026/0184",
      "issue_date": "2026-03-14", "currency": "EUR", "total": 1234.56 },
    { "kind": "receipt", "merchant": "Padaria Central",
      "paid_at": "2026-03-15T09:12:00", "payment_method": "card", "total": 4.9 },
    { "kind": "id_card", "full_name": "A. Silva", "document_number": "12345678",
      "nationality": "PT", "date_of_birth": "1990-06-02", "expires_at": "2031-06-02" }
  ]
}

That replaces the classification step, the routing step and three separate parser calls with one request. The discriminator is just a field with an enum, so your TypeScript union narrows on it directly:

for (const doc of result.documents) {
  if (doc.kind === "invoice") await saveInvoice(doc)
  if (doc.kind === "receipt") await saveReceipt(doc)
  if (doc.kind === "id_card") await saveIdentity(doc)
}

anyOf, oneOf, $ref and $defs all work, so if your schemas already live in a shared definitions block you can point at them instead of inlining.

Mapping that back to the five steps

Step 1, classification. Handled by the union above, in the same pass that extracts the fields. No second model call, no routing code.

Step 2, input handling. Irrelevant here. PDFs are read visually, up to 200 pages in one job, so a photographed page and a software-generated one take the same path. DOCX, XLSX and CSV are handled too. No parser per format.

Steps 3 and 4, extraction and normalization. The rules do not go in your prompt, so you do not pay input tokens to restate them on every document. format on a field means the value is resolved after extraction, in code: ISO dates, ISO country and currency codes, E.164 phone numbers, IBANs verified with their checksum, and QR or barcode payloads read by a decoder rather than by the model. Anything that does not resolve comes back exactly as printed instead of being guessed. Full table in Schemas and formats.

Derived values are computed by the runtime rather than by the model, and multipleOf: 0.01 puts money on two decimals rather than 110.60000000000001.

Step 5, durability. Every job returns an id immediately, retries transient failures with backoff without duplicating work, and delivers to a webhook. A failed job refunds its hold in full, so a failure costs nothing.

The two things you should still do

Not everything is ours to solve, and pretending otherwise would cost you money.

Check the document against itself. Line items should sum to the subtotal, subtotal plus tax should equal the total, a due date should not precede an issue date. Documents are highly redundant, and that redundancy is free error detection that needs no model:

const itemSum = invoice.line_items.reduce((sum, item) => sum + item.amount, 0)
if (Math.abs(itemSum - invoice.subtotal) > 0.01) await flagForReview(invoice)

**Treat null as information.** A field that is genuinely absent comes back null rather than filled with something plausible, which is why the required list in that schema is short. Route nulls to a human instead of forcing a value out of the extraction.

Multiple pages, multiple files, one job

sources is an array, and everything in it is read as one document set. Front and back of an ID card, a stapled invoice with its delivery note, a folder of receipts: send them together and the schema is filled from all of them.

{
  "sources": [
    { "type": "document", "data": "data:application/pdf;base64,JVBERi0..." },
    { "type": "image",    "data": "data:image/jpeg;base64,/9j/4AAQ..." }
  ],
  "schema": { "…": "the schema above" }
}

The short version

1. Write one schema that describes every shape you accept, with a kind enum per branch.
2. Put format on the fields that must be typed rather than transcribed.
3. Send whatever arrives, in one job.
4. Check the arithmetic, route the nulls.

That is the whole pipeline. The quickstart runs it in a few minutes, and the invoice walkthrough goes field by field on a single document type.