Under the Hood of IProcessor: Parsing 60+ Vendor Invoice Formats with an Azure WebJob
Part 2 of 2. Part 1 introduced the product and the web application. This post covers the invoice-parsing engine (IProcessorPdfParser), the technology stack, and how we ship it.
The problem
GMD Stores buys from more than 60 vendors. Each one sends invoices as PDF attachments, and no two layouts are alike. Some are clean text PDFs, some are scanned, some are XPS files, and a few vendors have two completely different templates depending on which office issued the invoice. The goal was simple to state and hard to do: read every invoice that arrives in the mailbox, extract every line item, and save it to SQL Server with zero manual typing.
IProcessorPdfParser: the always-on mailbox worker
IProcessorPdfParser is a .NET Framework console application deployed as a continuous Azure WebJob next to the website. It runs a single instance (guarded by a named Mutex so a second copy exits immediately) and goes through this loop:
Step 0: Connect to the mailbox
The worker signs in to an Office 365 mailbox through Exchange Web Services (EWS). Authentication uses MSAL (Microsoft.Identity.Client) with OAuth 2.0 client credentials, because Microsoft retired basic authentication for Exchange Online. On startup it first sweeps any unread messages it missed, then opens a streaming subscription so new mail is pushed to it instead of polled.
Step 1 and 2: Download attachments
For each new email the worker downloads PDF and XPS attachments to a working folder. XPS files are converted to PDF with PdfSharp.Xps so the rest of the pipeline sees one format.
Step 3: Identify the vendor
Vendor detection is two-layered:
- Keyword match: the first page is extracted to text (EvoPdf.PdfToText, with iTextSharp as fallback) and compared against the vendor keyword table that admins maintain in the web app.
- Docparser API: if keywords are inconclusive, the file is sent to the Docparser cloud service through our IProcessorDocParser library, which returns a structured JSON result including the vendor it matched.
If neither method finds a vendor the file is moved to a VendorNotFound folder and an alert email goes to the admins with the PDF attached.
Step 4 and 5: Parse line items
Once the vendor is known, a vendor-specific parser class takes over. There are more than 60 of them, all implementing a common InvoiceParser contract, and they share an ItemTableContentStrategy that knows how to walk a table given the header configuration stored in the database. The parser produces:
- Invoice header: number, date, PO number, store name, ship-to address, totals
- Line items: item number, UPC, description, quantity, case pack, unit cost, extended cost
The worker then validates the result. The sum of line items must match the invoice total within a tolerance, and the store name must resolve to a known store. A mismatch stops the invoice from being saved, moves the file to a review folder and emails the reviewers with the exact discrepancy. This single rule catches most parsing regressions the day a vendor changes its template.
Step 6: Persist and notify
Valid invoices are written to SQL Server (Entity Framework 6 for the web schema, DevExpress XPO for a few legacy tables), the original PDF is uploaded to Azure Blob Storage through the IProcessorStorageService library, and a confirmation email is sent through Exchange. Every step is logged to Application Insights with the invoice file name, so a support engineer can trace one invoice end to end.
Resilience
- Polly wraps the EWS and Blob calls with exponential-backoff retries.
- A configurable timeout keeps the WebJob process alive between emails; if the streaming subscription dies, the WebJob host restarts it.
- Admins can check the worker's heartbeat and trigger a restart from the website (the StartIParser and CheckIParser actions) without opening the Azure portal.
- A separate small tool, ReadEmailOfficeOAuth2 (.NET Core 3.1 + MailKit), reads the same mailbox over IMAP with OAuth 2.0. We use it to verify tenant credentials and as a fallback path if EWS is ever deprecated.
Solution layout
| Project | Type | Purpose |
|---|---|---|
| IProcessorWebsite | ASP.NET MVC 5 (.NET 4.8) | The web app: invoices, inventory, warehouse, reports, admin |
| IProcessorPdfParser | Console app / Azure WebJob (.NET 4.8) | Mailbox listener and invoice parser |
| IProcessorDocParser | Class library (.NET 4.6.2) | Client for the Docparser API and result models |
| IProcessorStorageService | Class library (.NET 4.8) | Azure Blob Storage and Azure AD helpers |
| ReadEmailOfficeOAuth2 | Console app (.NET Core 3.1) | IMAP + OAuth 2.0 mailbox reader and diagnostics |
| Database | SQL scripts | Tables, views, stored procedures and functions, source-controlled |
Technology stack
Backend
- C# on .NET Framework 4.8 for the website, worker and services; .NET Core 3.1 for the mail diagnostics tool
- ASP.NET MVC 5 with Razor views, ASP.NET Identity for authentication and role-based authorization
- Entity Framework 6 (database-first, EDMX) for data access; DevExpress XPO in the worker for legacy tables
- Newtonsoft.Json, Optional (functional option types), Polly for retries
Documents and files
- iTextSharp, PdfSharp, Spire.PDF and EvoPdf for text extraction, PDF to HTML and XPS to PDF conversion
- Docparser cloud API for vendors with irregular layouts
- EPPlus and ExcelDataReader for Excel import and export; Office Interop for a few legacy report templates
- Microsoft Exchange Web Services (Microsoft.Exchange.WebServices) with MSAL OAuth 2.0 for reading and sending mail
- MailKit for the IMAP diagnostics tool
- Outbound notifications were moved from SendGrid to Exchange so all mail comes from the company domain
Frontend
- Bootstrap 4 admin theme, responsive for warehouse tablets
- jQuery 3.5, jQuery Validation and Unobtrusive AJAX, ASP.NET bundling and minification
- Zebra BrowserPrint JavaScript SDK for direct-to-printer labels and RFID tags
- Chart.js, Highcharts, Morris and Flot for dashboard graphs
Database
- Azure SQL Database (SQL Server). The whole schema lives in the repo as SQL scripts: roughly 60 tables (Invoice, Item, DBInventory, MasterList, Store, Vendor, ContainerItems, PickAndPackResults, StationaryRFID, header and keyword configuration with history tables), plus views, stored procedures and functions for reports.
Cloud and operations
- Azure App Service with deployment slots (staging, disaster-recovery, production)
- Azure WebJobs for the continuous parser
- Azure Blob Storage for invoice PDFs and images
- Application Insights for logging and request telemetry
- Azure Key Vault for secrets; connection strings are injected at deploy time, never committed
CI/CD with GitLab and Azure slots
The repository is on GitLab and the pipeline runs on Windows runners with MSBuild 2019 Build Tools. Deployment uses Web Deploy publish profiles for both the website and the WebJob. The branch strategy maps directly to environments:
| Branch | Target | Who checks |
|---|---|---|
| Any feature branch | Staging-DR slot | The developer verifies their own change in a real Azure environment |
| dev | Staging slot | Team integration testing after merge request review |
| master | Production-DR slot | Support team and an end user do smoke tests |
| Release | Slot swap into Production | Zero downtime; swapping back is the rollback |
Because the WebJob is deployed together with the site, the parser and the web app are always the same version, which avoids schema mismatches between the two.
Lessons learned
- Put vendor layout knowledge in data, not code. The header and keyword tables let admins adapt to a changed invoice template in minutes. Only truly new vendors need a new parser class.
- Validate totals, always. The line-items-must-equal-total rule is the cheapest and most effective guard against silent parsing errors.
- Plan for authentication changes. Moving from basic auth to OAuth 2.0 on Exchange was the largest unplanned piece of work. Keeping a second mail path (IMAP + MailKit) ready made it low risk.
- Deploy to a slot, then swap. Warehouse staff work in shifts around the clock. Slot swaps gave us releases with no visible downtime and instant rollback.
Questions about the architecture or the parser design? Leave a comment below.
Comments
Post a Comment