How a tool-calling loop, hybrid retrieval, and tracked citations turn a chatbot into an answer you can verify

This article is the technical pair for What a good AI answer actually looks like. The business pair makes the case for grounded, agentic answers; this one is about how that answer gets built.

It also builds on The architecture of AI-native document management software, and on the published pieces about the search layer and the indexing pipeline.

---

When one search isn’t enough

The demo version of document chat is a straight line. The user types a question, the system rewrites it once, runs one search, hands whatever came back to the model, and prints the reply. It demos beautifully on three clean PDFs. Then someone asks a real question, the kind with three parts and a number in it, and the straight line returns a confident paragraph that’s quietly wrong, because a single search can’t answer a three-part question and a language model can’t be trusted to do the arithmetic.

The fix is to hand the model a few tools and let it decide how to use them, which is what turns one-shot retrieval into agentic RAG. Instead of running a fixed sequence, the system gives the model a small toolbelt and a loop, and the model chooses which tool to call, with what input, in what order, and how many times. The rest of this article is how that loop is wired, how each tool behaves, and how every claim in the final answer ends up carrying a link back to the page it came from.

Where this sits in the system

The chat box is the visible tip of a stack the user never sees. It reads from the search layer, which only contains what the indexing pipeline decided was worth making searchable, narrowed by the tags applied upstream. The orchestration runs on the Azure OpenAI Responses API, which is built around tool calling, so the same call that asks the model for an answer also tells it which tools it may reach for. Everything in this article assumes those lower layers exist and work; the chat layer adds judgment on top of them, and it can’t add trust the layers below never made possible.

Why the fixed pipeline breaks

The naive implementation fails in ways that have nothing to do with model quality.

It runs exactly one search per question, so “compare the payment terms in our last three supplier contracts” gets answered from whatever one query happened to surface, and the model writes a fluent comparison across documents it never retrieved together. It feeds arithmetic straight to the model, so totals and pro-rated figures come back looking plausible and landing slightly off, because predicting the next token is a different activity from calculating. It sends the raw user text into the search engine, typos and vague pronouns included, so the retrieval is aimed at noise. And it treats citations as a formatting step at the end, pasting a reference list under an answer where no individual claim points to a specific page, which leaves the reader to re-read the source to check anything.

Each of these has a concrete fix, and the fixes only fit together inside a loop where the model is in control of the steps.

The agentic RAG loop

The model is given three tools and told it may use them as it sees fit.

The first is a query-rewrite tool. The model itself produces the corrected query as the tool’s argument, fixing spelling, grammar, and informal phrasing before any search runs, so the rewrite is just the model cleaning up its own input. The second is a document-search tool that runs retrieval against the search index with the user’s container and tag filters applied. The third is a sandboxed code tool, the built-in code interpreter, which runs real Python for any calculation the model would otherwise guess at.

The orchestration is a loop. The backend makes the first model call with all three tools attached and tool_choice="auto", which lets the model decide whether to call a tool or answer directly. If the response contains tool calls, the backend runs each one, packages the results as tool-output items, and makes another model call carrying those results. That next call is chained to the one that requested the tools by passing its response id, so the model keeps its full reasoning and tool history without the backend resending the transcript. The loop repeats until the model returns an answer with no further tool calls, or until a hard iteration cap stops it.

tools = [
    rewrite_query_tool,     # model returns the corrected query
    search_documents_tool,  # hybrid retrieval with the user's filters
    {"type": "code_interpreter", "container": {"type": "auto"}},
]

response = client.responses.create(
    model=deployment,
    input=user_turn,
    instructions=system_prompt,
    tools=tools,
    tool_choice="auto",
)

while has_tool_calls(response) and iteration < max_iterations:
    iteration += 1
    outputs = run_tools(extract_tool_calls(response))   # rewrite / search / code
    response = client.responses.create(
        input=outputs,                      # tool results fed back in
        instructions=system_prompt,
        tools=tools,
        tool_choice="auto",
        previous_response_id=response.id,   # chain to the call that asked
    )

The iteration cap is the safety net that a tool loop needs, since a model that keeps deciding it wants one more search would otherwise run forever. When the cap is reached the backend stops and answers with whatever has been gathered, and it logs that the cap was hit so the behavior is visible to operators.

The decisions that make it work

A loop with three tools is the skeleton. The behavior that makes the answers trustworthy lives in how each tool is handled.

Rewrite first, then search once or many times

The model decides the sequence. When the user’s question has spelling mistakes, typos, or vague phrasing, it calls the rewrite tool first and gets back a cleaned version of the whole question. After that it picks one of two paths based on complexity.

For a single-topic question, it runs one search aimed at that topic. The search query should be the cleaned question, or a focused variant of it. The backend adds a small safety net here: if the model requested a rewrite and exactly one search in the same tool batch, but the search argument still carries the messy original wording, the backend replaces it with the rewritten text before retrieval runs. That way a typo fix can’t be lost because the model forgot to wire the rewrite into the search call.

For a multi-part question, the model decomposes it into several searches, each with its own focused sub-query (for example three separate lookups for three contracts). Those sub-queries are meant to be different from each other and from the full rewritten question, so the backend never substitutes the rewritten whole question into any of them. Pasting the full rewrite into every search in a decomposition batch would run the same broad query three times, which defeats the point of splitting the question.

Take “Compare payment terms in our last three supplier contracts”. The wording is already clean, so the model skips rewrite entirely and calls the search tool three times in one batch, each time with a different query aimed at one contract (for example “payment terms Supplier A contract”, “payment terms Supplier B contract”, “payment terms Supplier C contract”). That is zero rewrites and three searches. If the same question arrived with typos (“Compare paymnt terms in our last three supplier contracs”), the model would call rewrite once to clean the whole sentence, then still call search three times with three separate sub-queries it writes itself. That is one rewrite and three searches. Rewrite always works on the full user question; it never runs once per search, and the backend never splits a rewrite into sub-queries for you.

The intended flow is rewrite when the question needs cleaning, then either one search that uses the cleaned wording or several searches each aimed at a different part. The backend only forces the rewrite into the search step in the single-search case.

Retrieval that’s bounded and deduplicated

Every search call runs hybrid retrieval, combining keyword matching with vector search so the query matches on both wording and meaning. Before the results reach the model they pass through several bounds. A container filter (a folder holding all documents for a specific group, such as a department) is always applied and the search is rejected outright if one can’t be built, so a user can never retrieve from a container they aren’t allowed to see. The user’s tag selections (business context labels the organization applies to documents, such as supplier, document type, or region) become additional filters on top of that. A score-gap cutoff then trims the tail, keeping results only while their relevance stays within a band of the top hit, so a weak match never pads the context just to fill a slot.

Because the model can search several times in one turn, the backend tracks state across those calls. It deduplicates chunks it has already returned this turn, and it enforces a per-turn chunk budget, so ten searches can’t quietly balloon the context window and the bill with it. When the budget is spent, further searches return a short note saying the limit was reached rather than more content.

Context that travels with the chunk

Each retrieved chunk arrives already carrying the short document-level summary the indexing pipeline wrote for it, and the backend formats that summary into the source block the model reads. This is where the per-chunk context investment from indexing pays off, because a paragraph that says “as described in the section above” is usable on its own once its context line travels with it. The formatted block also carries the fields that make grounding possible later.

REF: S1
TITLE: supplier-agreement.pdf
PAGE: 12
CONTAINER: contracts
CONTEXT: Section 7 of the 2025 master services agreement with Supplier X.
CONTENT: The liability of either party shall not exceed ...

Each source gets a stable reference id like S1, and the ids keep counting up across multiple searches in the same turn so two search calls never hand back the same id for different documents. That stable id is the thread the citation system pulls on at the end.

Math goes to code, not the model

When a question needs a real calculation, totaling line items or working out a pro-ration, the model hands it to the sandboxed code tool, which computes the number in Python and returns it. The model still narrates a short, human-readable version of the calculation, but the figure itself comes from code that ran, not from a token the model predicted. Sandbox paths and any mention of the execution environment are kept out of the reply, so the user sees the result and the reasoning, never the plumbing.

Turning a reference into a link you can click

The model is instructed to mark every fact it draws from a source with that source’s reference id, so the raw answer is full of [S1], [S2] style markers. Turning those into something a person can use is a backend step. Each reference id maps back to its document title, page, container, and chunk kind, and the backend rewrites every marker into a readable label and a clickable citation that opens the document at the right page.

That step also cleans up the details that would otherwise erode trust. Converted Office files are stored as PDFs in the canonical copy, so a spreadsheet shows up internally as orders.xlsx.pdf; the backend strips that trailing suffix back to orders.xlsx for display. Citations are deduplicated by document and page so the same page cited five times collapses to one reference, and page numbers are dropped for tabular sources where a page number means nothing. Alongside the rewritten text, the backend emits a list of unique citations and a map from each marker’s position to its citation, which is what lets the interface render the inline links in the exact spots the model placed them.

Keeping the thread across turns

A real exchange is rarely one question. Continuity here rests on the same response-id chaining the tool loop uses. The Responses API holds the state of each response on the server, so the backend stores the id of each turn’s final response in the chat record and replays it as the starting point of the next turn. A follow-up like “and what about the renewal clause?” then arrives with the whole prior thread already attached, including the earlier searches and citations, without the backend resending the transcript.

The conversation record lives in Azure Cosmos DB, partitioned by user, with each turn storing the user message, the assistant answer, its citations, and the response metadata. Two answer profiles share all of this and differ only in reasoning effort, a faster, cheaper one for everyday questions and a slower, more thorough one for hard ones, chosen per turn and remembered for the conversation.

Reality check

This costs more than a single model call, and the cost is worth naming rather than hiding. An agentic answer can involve a rewrite, several searches, a calculation, and the final composition, which is multiple back-and-forth steps where the demo had one, so it’s slower and more expensive per question. The per-turn chunk budget exists precisely to keep that cost bounded, by capping how much retrieved content a single turn can pull in no matter how many times the model searches.

The server-side conversation state has a limit too. The Responses API retains a response for thirty days, so a conversation left idle past that window loses its chained context, and the backend detects the expired-history error and tells the user to start a new chat rather than failing silently. Because those responses persist on the provider’s side, deleting a chat also collects every response id it accumulated, final answers and intermediate tool calls alike, so the stored state can be cleaned up rather than left dangling, which keeps the data lifecycle honest.

And grounding is only ever as good as the layers beneath it. The answer can cite a page accurately only because indexing preserved which page each chunk came from, and it can limit search to the right documents only because the tags upstream are clean enough to filter on. The loop adds judgment, the citation step lets every claim be checked against its page, but neither can manufacture trust that the storage, indexing, and tagging layers didn’t already earn.

Conclusion

An agentic answer is best understood as a small program the model writes for each question. Given a rewrite tool, a search tool, and a code tool, plus a loop that keeps handing results back until it’s done, the model decomposes the question, gathers what it needs, computes what it must, and composes a reply, and the backend’s job is to run that program safely, bound its cost, and make every claim checkable.

That last part is the real product. Grounding here is tracked end to end, from the numbered source the model reads, through the marker it writes on each claim, to the clickable link that opens the right page, and it’s persisted and cleanable alongside the conversation rather than reconstructed for show. A system built this way stops being graded on how its answers sound and starts being graded on whether someone can click through to the page a claim came from, which is the difference between a demo that impresses once and a tool people trust with real decisions.

---

Ideas, opinions, and tone are mine. AI helped with the language.

For more articles on AI-native document management visit the pialgorithms blog.

---

pialgorithms | document management software | ai engineering services