Skip to content

0004 — Transit extraction uses a deterministic PdfPig skeleton and two AI calls

Status: accepted

Context

A Transit Accompanying Document can carry hundreds of goods items — 281 on the case that forced this. Asking the model for the whole declaration in one response means asking it to emit hundreds of JSON objects, and past a certain size it truncates. Truncation is the worst possible failure mode here: the JSON parses, the declaration looks complete, and items are simply missing from the end. Nothing errors. A declaration filed short of goods is a customs problem, not a software one.

The item numbers and package counts are also the part the model is least needed for. They are laid out positionally in the PDF's content stream, so they can be read deterministically — and a deterministic read cannot hallucinate an item number or lose one.

Decision

Five steps, in ExtractionService:

  1. PdfPig builds a skeleton — item numbers and packages — straight from the PDF content stream. No AI, no network.
  2. Document Intelligence extracts page-level OCR text.
  3. AI header call — page 1 OCR → header and global fields only.
  4. AI items call — pages 2..N OCR plus the skeleton → fills description, commodity code, gross mass, net mass and the previous/supplementary documents into each skeleton item.
  5. Merge header and items into the raw JSON.

The items prompt states that the skeleton's numbers are pre-filled and must not be changed, and the header prompt is told to return no item-level fields.

Consequences

  • The item count is decided by PdfPig, not by the model. That is the point: the model fills in fields for a list whose length is already fixed, so a short response leaves fields empty rather than silently dropping goods. Empty fields are visible in review; missing items are not.
  • Packages are never asked for, so the model cannot get them wrong.
  • Two AI calls per transit dossier instead of one — more latency and more cost, deliberately traded for not truncating.
  • PdfPig's skeleton is now load-bearing. If a TAD layout changes such that PdfPig reads the item numbers wrongly, the whole extraction is wrong in a way the AI cannot correct, because the AI is explicitly told not to. Any change to TransitPackageParser should be checked against a large multi-page TAD, not a two-item one.
  • This is the primary transit path, not a fallback for big documents — so it is exercised on every transit dossier, which is what keeps it working.

Amended 2026-09-07 — the skeleton is enforced, and it knows two layouts

The consequence above about the skeleton being load-bearing had a hole: when PdfPig read nothing, the pipeline fell through to the model building the whole list itself, silently. On TUF's own NCTS5 T1s the parser never engaged — it demanded the two item-number columns to be equal, and on that layout the goods item number restarts inside every previous-document group — so a 48-page, 127-item TAD came back once with 129 items (two numbers duplicated) and once with 127 items of which two had no commodity code. The 33-page version of the same document had been fine. Nothing errored either time.

Three changes:

  • TransitPackageParser reads both layouts. Layout B is anchored on each item's tail — gross mass, departure means, six-digit commodity code, net mass — and validated as a whole (the k-th item must carry declaration number k, goods numbers restart at 1 or count up by one). It also reads the commodity code and masses, which the model previously supplied. Anything inconsistent returns null: the parser is allowed to be right or silent, never partial.
  • The item list is enforced after the merge, not hoped for. goods_items is rebuilt from the skeleton: one item per document item, in order, with PdfPig's packages, code and masses, and the model's description and documents attached by declaration number (the only unique one). A model item the document does not have is dropped; a document item the model omitted is emitted with empty fields, visible in review.
  • Reconciled before presenting. A header item count that disagrees with the document is logged; an item left without a commodity code fails the extraction with the item numbers in the message, so it lands as Failed rather than in Review looking complete.

The fixture tests/CustomsHive.Tests/TestData/Transit/tad-127-items.pdfpig.txt is that document's content-stream text, anonymised, with both shapes in it. Any change to the parser must keep it at 127.