CustomsHive

Belgian customs declaration processing tool. Accepts invoice PDFs, packing list PDFs, and/or Excel files; extracts structured goods data via AI; allows review and correction; then generates compliant IDMS/NCTS/AES XML declarations for submission. External systems can push declarations directly via REST API or Azure Service Bus using the canonical ingestion contract.

Stack

• ASP.NET Core 10 Blazor Server (fully converted from Razor Pages) — EF Core + SQL Server; extraction jobs (ExtractionJobs) and outbound declarations (OutboxMessages) are durable SQL-backed queues drained by in-process workers, so a restart resumes rather than discards
• AI extraction — Azure Document Intelligence (the OCR) + Azure OpenAI (classification + structured extraction on the OCR text); transit uses a deterministic PdfPig skeleton plus two AI calls; a legacy GPT-vision path remains behind "Process with AI"
• Auth — Microsoft Entra ID (OIDC); role-based on Admins, SuperUsers and Users, which is what the app registration defines and what the authorisation policies require. Two traps, neither of them currently biting:
• The claims transformation also maps Beheerder → Admins and Hoofdgebruiker → Users. No such roles exist in Entra, so that mapping is vestigial and never fires. Note it maps Hoofdgebruiker to Users rather than SuperUsers, and has no entry for Gebruiker at all — so reviving the Dutch names without fixing the mapping would silently demote one and lock the other out.
• Entra also defines four team roles — CustomsAntwerpen, CustomsAalst, CustomsRekkem, CustomsGent — matching the four CustomsTeams rows. The application does not read them. A dossier's customs team is chosen from a dropdown, per dossier. A user assigned only a team role holds nothing the policies recognise and gets no access.
• XML generation — custom generators per message type (no third-party library)
• Reference data — Tarbel/UN-LOCODE/code lists in application databases and services
• Ingestion — CustomsHive.Module.Ingestion: REST API endpoint + ServiceBus receiver; both use the same CanonicalDeclaration contract

Core domain concepts

Dossier

Central entity. Each dossier represents one shipment/declaration file. Key fields: - Ucr — unique customs reference - Lrn — column exists; the LRN is assigned by Descartes after filing and the app never receives it, so this is not populated - ContainerNumber, SupplierCode - Regime — "IM" (import), "EX" (export), "T1"/"T2" (transit) - Status — manual: Queued → Processing → Review → Approved → Ready → Submitted (or Failed); ingested: starts at Review. Submitted means "handed to the Service Bus queue"; Descartes returns no outcome. See customs.md for the exact transition table - Source — Manual (uploaded via UI), ApiIngestion (via POST /api/declarations), ServiceBusIngestion (via canonical ServiceBus queue) - IngestionPartner — name of the sending system for ingested dossiers (API key name or ServiceBus Partner property); null for manual dossiers - CreatedBy — username (manual) or partner name (ingested) - RawExtraction / CorrectedData — AI output + user corrections stored as JSON - InvoicePdfPath, PackingPdfPath, XlsxPath — uploaded source documents

Client

Importer/client profile. Pre-filled into declarations when a dossier is created. Key fields: - Code — short identifier (e.g. "SKCH") - Name, IdentificationNumber (EORI) - Address: StreetAndNumber (max 70), Postcode (max 17), City (max 35), Country (ISO2, max 2) - Authorisation references: Fr1 (BTW importeur / FR1), AuthC503, Ref4007, Et14000 - DefaultProcedure — default procedure code (Standaard Regeling) - RefCBAM — CBAM account number (used when Y128 applies); AirTransportRatePerKg / AirTransportMinimum — client-specific air "Transport BE" rate and floor - IsCustomsClient — whether the client appears in the dossier client picker. true for manually created clients and for clients auto-created by an incoming declaration; false for clients imported from Business Central masterdata, which is everyone the business invoices rather than the subset that clears customs. Promoted by hand on /Clients → All clients.

CustomsDeclaration

Per-dossier declaration metadata captured at submission time: - Reference documents: BillOfLadingRef (N337), OriginCertRef / OriginCertDate (N935) - Transport: ContainerNumber, ContainerIndicator, DestinationCountry, CountryOfDispatch - SupervisingCustomsOfficeRef (8-char office code) - NatureOfTransaction, UseH2B (H1B = standard import, H2B = customs warehouse / procedure 71) - Static company fields (sender GLN, company EORI, authorisations) read from AppSettings

Extraction

Each extraction attempt is stored as ExtractionRecord with: - ExtractorType — pdf_di_hybrid_invoice / _packing_list / _transit, xlsx_*, or the legacy pdf_vision_* - AdiRawResponse — the Document Intelligence OCR JSON, cached and reused on reprocess - RawResponse — the raw AI response; ExtractedData — the same after fence-stripping and best-effort JSON extraction - PdfPigRawText — the deterministic transit skeleton source - token counts, page count and an estimated cost per attempt (summed on /Admin/Costs)

Supported XML message types

Implemented XML generators (IE415BXmlGenerator, CC015CXmlGenerator, CC515CXmlGenerator) read corrected data + declaration metadata and produce schema-valid XML per the Belgian IDMS/NCTS/AES XSDs.

Key workflows

1. New dossier (manual)

/NewDossier — upload invoice PDF + packing list PDF (and/or XLSX), select regime, select or create client inline, set UCR/LRN/container.

2. AI extraction

ProcessingWorker claims jobs from the durable ExtractionJobs queue (up to Processing:MaxConcurrentJobs at once, 30-minute visibility timeout, three attempts). Each PDF goes to Document Intelligence for OCR; the OCR text goes to Azure OpenAI with the prompt for its document type; the JSON comes back as RawExtraction on the dossier, which moves to Review. Multiple extraction attempts are kept and can be compared.

3. Review & correction

• IM: /Dossiers/{id}/ReviewImport — review extracted invoice lines, set HS codes, values, quantities
• EX: /Dossiers/{id}/ReviewExport — review export goods lines
• T1/T2: /Dossiers/{id}/ReviewTransit — review transit goods, raw JSON panel for debugging
• Goods breakdown pages: /Dossiers/{id}/GoodsBreakdownImport, /Dossiers/{id}/GoodsBreakdownTransit
• Transport costs: /Dossiers/{id}/TransportCosts (standalone calculator at /Tools/TransportCosts)

4. Declaration

/Dossiers/{id}/DeclareImport / DeclareExport / DeclareTransit — Beheerder/Hoofdgebruiker fills in declaration-specific fields (B/L ref, origin cert, customs office, container indicator, etc.) and generates XML (IM, EX, T1/T2).

5. Clients

/Clients — CRUD for client profiles. Address + authorisation references populated here are auto-loaded into new dossiers.

6. Canonical ingestion (automated)

External systems (Qargo, VDM, etc.) push CanonicalDeclaration payloads via: - POST /api/declarations — REST API, authenticated via X-Api-Key header or Azure AD Bearer token with Ingestion.Write role - Azure Service Bus — canonical queue (ServiceBus:Ingestion:*), JSON or XML body. The Partner application property identifies the sender and is required; MessageType selects Declaration (the default) or Client masterdata.

The ingestion service resolves the client (by client_code shortcut, or EORI lookup, or auto-creates one), routes to the correct adapter (CanonicalToImportAdapter / CanonicalToExportAdapter / CanonicalToTransitAdapter), and creates a dossier in Review status with RequiresReview = true. A transit declaration carrying auto_dispatch is filed without review; every other declaration type carrying it is rejected.

See docs/ingestion-api.md for the full integration guide.

AI configuration

Two Azure services are used together for PDF extraction:

Both services share the same app-registration credentials (Azure:TenantId, Azure:ClientId, Azure:ClientSecret) for keyless auth. See configuration.md for full key reference.

Extraction pipeline

flowchart TD
    subgraph Input
        PDF(["Invoice / Packing List PDF"])
        XLSX(["Excel XLSX"])
    end

    PDF --> ADI["Azure Document Intelligence\nprebuilt-layout OCR"]
    XLSX --> XP["XlsxProcessor\n(direct parse)"]

    ADI -->|OCR text| CL["Azure OpenAI\nclassify_document prompt"]
    CL -->|invoice / packing_list| EX["Azure OpenAI\nextract_invoice / extract_packing_list"]
    EX -->|structured JSON| DB[(ExtractionRecord)]
    XP -->|structured JSON| DB

    subgraph Transit
        TPDF(["Transit TAD PDF"])
        TPDF --> TADI["Azure Document Intelligence\nprebuilt-layout OCR"]
        TADI -->|OCR text| TEX["Azure OpenAI\nextract_transit prompt"]
        TEX -->|structured JSON| DB
    end

XLSX files are parsed directly without AI — no OCR or LLM call needed.

Transit PDFs skip classification: PdfPig builds a deterministic item skeleton, then one AI call fills the header from page 1 and one fills the items from the continuation pages (ADR-0004). The GPT-vision path (PDF → PNG → Azure OpenAI vision) is legacy: it remains as a one-page fallback for the transit header and behind the "Process with AI" button.

Prompts are stored in the Prompts table (DB) and editable via /Admin/Prompts.

Reference data

• Tarbel — Belgian tariff/nomenclature data; TarbelService resolves HS codes, descriptions, applicable VAT
• Country codes — CountryCodeService resolves country names/codes from the Tarbel GeographicalArea table
• Locode — UN/LOCODE lookup via LocodeService; single endpoint GET /api/locodes/search?q=&limit=
• EU code lists — CodeListService, backed by a daily sync of DG TAXUD's CS/RD2 publication into the tariff database. The JSON embedded in the application is the fallback floor, not the source: it was photographed in 2022, and a still copy of a moving list is wrong in both directions — it refuses codes that exist and offers codes that no longer do.
• Customs offices — CustomsOfficeService, from the same publication, with the roles (export, exit, departure, destination, transit, supervising) that say what each office may act as.
• Browsing them — /Integrations/ReferenceData shows what the pickers offer and whether it is synced or the embedded floor. It is the place to check whether a code exists before concluding a declarant typed it wrong.

One thing worth knowing before reading a count: CS/RD2 publishes the same list separately per business domain and they genuinely differ, so a supporting-document list is 333 codes for an export declaration and 294 for an import one.

SMF XSD validation

SMF wrapper validation can run against bundled XSD files before submit/return. In Docker images built from this repo, XSD files are available at /app/schemas/smf and runtime defaults include:

• Descartes__Smf__ValidateXsd=true
• Descartes__Smf__XsdFolder=/app/schemas/smf

Quick start (Docker)

cp .env.example .env
# fill in .env
docker compose up -d

Docker images are published to ghcr.io/rousseauxy/customshive. Tags follow docker-x.y.z convention on GitHub → image tag x.y.z.

See docs/configuration.md for all environment variables.

Deployment

Both environments — SRXACT (dv) and OCI (live, serving app.customshive.eu) — run the container image. Push a docker-* tag → GitHub Actions builds and pushes to ghcr.io. dv tracks :latest; OCI pins an explicit version, so rolling it forward means editing the pin, not just pulling. See the internal operator runbook for the procedure.

The container image is the only way this ships. The IIS and Azure DevOps path was deleted in September 2026 (B-16) — the workflow's v* publish, azure-pipelines.yml and Deploy-IIS.ps1 are gone, along with the setup instructions. Nothing had run on it since the move to containers.

See docs/deployment.md for full setup instructions.

Docs