Building registry-based extraction for any document type
Two-level discovery and a tenant allowlist in front of lookup are how a new document type, or a new field set of an old type, lands without touching the engine.
This article is the technical pair for One extraction engine, any document type. The business pair makes the case for treating a new document type as configuration; this one is about the registry and generic engine that make that real.
It also builds on How a document workflow engine differs from a chat service and on The architecture of AI-native document management software.
The second type is the real test
Shipping a contract extractor is the easy win. The hard question arrives the week finance asks for invoices, and the architecture either absorbs that request as a small folder or forces another project with its own endpoints, its own client, and a fresh slice of the roadmap. If the second document type needs new routes, the system already failed the test that decides whether extraction ever scales.
There’s a second test the week after. Another organization also extracts contracts, and their columns don’t match the first field list. If that request becomes a new public type, or a merge of both schemas, the registry still encoded the wrong axis.
This piece is how a registry-based extraction engine passes both tests. One process discovers families and named field sets at startup, one set of generic endpoints stays parameterized by public type only, and the shared engine runs the same retrieve-call-validate-persist path with no tenant or type if ladder.
Where this sits in the system
The previous article drew the line between a chat service and a workflow engine. This one goes into the workflow side’s first load-bearing part, the registry that turns “any document type” from a slogan into a folder convention. Schema-first validation and layout-assisted field locations get their own deep dives next; here the focus is two-level discovery, the tenant allowlist in front of lookup, the generic API, and the engine path those layers hang from.
Every extraction still reads from the same search layer the chat service uses. The registry doesn’t invent a second retrieval stack. It only decides which complete field set (schema, form layout, optional business rules, prompt) and which Cosmos container apply once the chunks for that one file are in hand.
Where things go wrong
The naive build wires the document type into the HTTP surface. Contracts get /contracts/extract. Invoices get /invoices/extract. Each handler imports its own schema (for the output shape), its own prompt file (for extraction instructions), and its own Cosmos client (for extracted data storage), and the “generic” bits are whatever someone remembered to copy. It feels tidy on day one because there’s only one type and every name matches.
The bill shows up on type two. You duplicate the contract stack, rename the paths, adjust the fields, and ship a second near-copy that starts drifting the moment either side gets a bugfix. By type five you’re maintaining five route tables, five clients, and five slightly different ideas of what “extract” means. A change to retrieval, audit shape, or concurrency has to be applied by hand in every copy, so it usually isn’t.
The other trap is the growing if ladder inside a single handler. if type == "contract" … elif type == "invoice" … looks cheaper than duplication because you keep one route file, but the type names still live in the core path, so every new document type means editing that ladder, reviewing the shared handler, and redeploying the whole extraction service before the type can run. A type that isn’t listed simply doesn’t exist to the API, and the ladder only gets longer as the business asks for more documents. Put if tenant == … next to it and you’ve encoded customers in the engine too. That keeps every new type, and every new field set, on the engineering roadmap instead of letting them land as configuration.
A type-only folder convention still fails the second customer. One schema named “contract” becomes a global field list. Two organizations that share the public type then share columns they shouldn’t, or you start forking types in the URL.
Registry-based extraction as the unit of extensibility
The way out is to stop encoding document types and tenants in the request handlers, and put them in a registry the process builds once at startup, with a tenant allowlist in front of every lookup.
On first use, the registry walks the extraction service’s package directory. Every subdirectory that isn’t private scaffolding (names starting with _, the shared core, cache folders) is treated as a candidate family, the public type. Family config holds the public name, the environment variable for that family’s Azure Cosmos DB container, and the chunk budget. Then discovery walks named field-set folders under each family. For each field set it imports a schema module, a UI layout module, and optionally a validators module. If those parts load and the UI keys match the schema keys, the registry stores an entry keyed by family plus field-set name. Folders that fail to import are logged and skipped so a broken field set can’t take down the whole process.
What lands in each entry is deliberately small:
- The family name (must match the public type and the folder).
- The field-set name.
- The environment variable that holds that family’s Cosmos container name.
- Pointers to the schema getter (required), the UI layout getter (required), and an optional business-rule validator.
- Enough naming convention to find the prompt profile for that family plus field set.
The prompt profile itself lives beside the other model profiles as a YAML file named for that pair. It carries the system instructions, the user template that receives the assembled source text, and model knobs such as reasoning effort and max output tokens. The schema module is mandatory because the engine refuses to call the model without an output shape. The UI module is mandatory because JSON Schema can’t express labels, widgets, or submit-blocking required sets. The validator module is optional because most structural correctness already sits in the schema, and cross-field business rules are the exception.
A template folder ships with the service so a new family starts from a known skeleton, without copying a live sibling. Discovery skips that template on purpose. The live families in this platform today are contracts and invoices; each currently ships one field set, registered the same way, and neither required a change to the HTTP layer when the second family landed.
The persistence client follows the family, not the field set. At init it asks the registry for every registered family, resolves each Cosmos container from its env var, and opens a container client per family under DefaultAzureCredential. Adding a family means provisioning its container and setting the env var; it doesn’t mean writing a new Cosmos client. Every write stamps the customer key and the field-set name so a later read can refuse another customer’s record.
One endpoint set, parameterized by type
Because the registry owns the type list, the API can stay generic. Every route takes the public document type as a path segment. The client isn’t in the URL. Before every extractor route, the signed tenant identifier from Microsoft Entra External ID is allowlisted against a customer row; that row selects the bound field set for the family. Unknown or missing tenant fails closed (403). A family that isn’t bound for that customer is rejected before any work starts (400).
GET /extractors/{document_type}/profile
GET /extractors/{document_type}/docs
GET /extractors/{document_type}/records
GET /extractors/{document_type}/extract-eligibility
POST /extractors/{document_type}/extract
PATCH /extractors/{document_type}/update
POST /extractors/{document_type}/submitThe handlers don’t branch on the type name or the tenant. They resolve the customer from the signed principal, validate the family against that customer’s bindings, then call the same engine and the same persistence client with the resolved field set passed through. The profile route returns UI metadata only (labels, widgets, required sets). It never returns the JSON Schema, the prompt, or the tenant id. List-docs joins storage documents tagged with that family to the extractor records for status. Extract-eligibility returns the documents that already hold saved form data so the UI can hide a redundant extract. Extract runs the engine and upserts the result. Update and submit write field edits and the final handoff, with optimistic concurrency so two people editing the same record don’t silently overwrite each other.
Unknown family in the path, or a family this tenant isn’t bound to, becomes a 400 with no search call and no model call. Unknown tenant becomes a 403. That early reject is the whole point of the allowlist sitting in front of the registry.
The shared engine path
Once the family and field set resolve, every extraction follows the same fixed path. The model doesn’t get to reorder the steps. The engine has no tenant switch and no type ladder; it only ever sees one resolved config.
Registry lookup. The engine asks the registry for the family plus field-set config. Missing pair or missing schema getter fails immediately with a clear error and writes nothing.
Focused retrieval. It builds a document-scoped search against the shared index, using the canonical blob name and the original container as the filter pair, and pulls up to the family’s chunk budget. Search failures propagate, because continuing with an empty source list would produce a junk extraction and a misleading audit record. Retrieved chunks are assembled into page-labeled source text.
One constrained model call. The prompt profile supplies the system instructions and the user template. The schema getter supplies the structured output format on the Azure OpenAI Responses API. The call sets store=False because this is a one-shot job with no next turn to chain to; the durable record is the engine’s own.
response = client.responses.create(
model=deployment,
input=[
{"role": "system", "content": instructions},
{"role": "user", "content": sources_text},
],
text={"format": {"type": "json_schema", **output_schema}},
store=False,
)Optional business rules. If the field set registered a validator, the engine runs it after parsing. Failures are logged for operators; structural enforcement already happened at the model boundary, so this layer is for rules a JSON Schema can’t express. The deeper treatment of schemas and validators is the next article.
Layout overlay (best effort). When a Document Intelligence layout sidecar exists for the file, the engine maps extracted values back to page regions for the UI highlight. Missing layout is non-fatal; extraction still succeeds. Field-level provenance gets its own deep dive later in the series.
Persist with audit. The HTTP layer upserts the structured fields onto the per-document Cosmos record in the family container, sets status to extracted, stores any field locations, and appends an audit entry that keeps what the model was shown and the full response it returned (with caller identity stripped). The record id is deterministic from family, container, and document, so repeated extractions of the same file converge on one shared record. Customer key and field-set name are stamped on the write; a record for another customer is refused on read.
How to tell a config change from a real build
This is the if-then test the registry exists to support.
If the request is a new public type that only needs a destination plus a first complete field set, it’s a family folder, a field-set folder, a prompt profile, and the env vars for its container and chunk budget. Optionally you add a validators module when cross-field rules are real. Restart (or let the next cold start) pick it up. The existing endpoints serve it with no handler edits.
If the request is different columns on an existing public type, it’s a new field-set folder under that family, a new prompt, and a new eval dataset. You bind the customer row to that field set. You don’t add a route, and you don’t merge schemas.
If the request is a new customer who can use a field set you already ship, it’s a binding; add their tenant to a customer row and point the family at the existing field set, with no new family.
If the type needs a processing step the shared spine can’t express, a different retrieval strategy, a multi-step model loop, a post-process that calls another system mid-flight, then you extend the engine or add a typed hook, and that should feel rare enough that everyone notices. The registry’s job is to make the first paths the default and the last path the exception you can see in the diff.
Concretely, teaching the platform invoices after contracts meant a new family folder, a field set, a YAML profile, a Cosmos container, and two app settings. It didn’t mean a second extract route. A second customer on contracts with different columns would mean a second field-set folder, not /extractors/contract_customer_b.
Reality check
Discovery at startup is cheap compared with the alternative of hand-wiring every type into the HTTP layer, but it isn’t free of operational rules. A field-set folder without a working schema or UI getter is skipped. A Cosmos env var that isn’t set fails client init for the whole extraction service, which is intentional, since a half-configured family is worse than a loud startup error. Prompt profiles follow a naming convention keyed off family plus field set, so a mismatch is a silent miss until the first extract.
Quality exams are per field set, not per family and not per user. Two customers who share a field set share the gold. Different columns mean a separate folder of cases and weights. A new customer with the same fields is a binding, not a new exam suite.
The registry also doesn’t remove the hard parts of extraction quality. A new field set can be live in a day and still need its instructions tuned against messy real documents before the field values are trustworthy. Configuration removes the engineering tax on adding types and customers; it doesn’t remove the work of getting a hard document family right.
Ownership stays with the business. Someone still has to decide which fields count for this tenant, what valid looks like, and where results may land. The registry just makes those decisions sit in small, reviewable artifacts instead of being buried inside five copied handlers or one global contract schema.
And two types are the break-even point. If you truly only ever need contracts for one customer, a hard-coded extractor is less machinery. The registry pays for itself when the queue of document types is real, and again when the second tenant can’t share the first field list, which is the usual case once the first extractor ships and the rest of the organization notices.
Conclusion
What makes an extraction system scale is that the second, third, and fiftieth public type cost about as much as describing their destination and first field set, and that the second customer on an old type costs a complete field set or a binding, with the public type still in the URL. The registry is how that property gets into the architecture, through two-level auto-discovery at startup, a tenant allowlist in front of lookup, one parameterized endpoint set, and a single engine path that reads resolved config instead of embedding type names or tenants in the handlers.
Everything after the lookup, the schema lock, the optional business rules, the per-document audit record, stays shared on purpose. The next article goes into that schema-first boundary in detail, because once field sets are cheap to register, the quality of each declared shape is what decides whether the extracted data is fit for another system to consume.
Ideas, opinions, and tone are mine. AI helped with the language.
For more articles on AI-native document management visit the pialgorithms blog.
pialgorithms | document management software | ai engineering services
