Inside Workflows: keeping agent research reproducible

Agents are good at deciding what to look at next. Financial research has to be repeatable and checkable. Those sound like opposing requirements, and most of our architecture comes from finding out that they are not.

September 15, 2026

Engineering, Agents, Workflows/12 minutes read

Someone has to be able to check financial research. Not “does the answer sound right”, but “show me the sentence in the transcript that supports this number”. Then, next quarter, the same analysis has to run again and produce something comparable. A credit review that changes method every time it runs is not a review.

Reproducibility and transparency are therefore hard requirements. They also sit awkwardly next to the thing agents are best at. The most valuable move a research agent makes is usually the one nobody scripted. It reads a transcript, notices that the guidance language has softened, and goes looking for the 8-K on its own. That judgement is where the quality comes from, and it is the first thing you give up if you decide a run has to be predictable.

That is the tension Workflows was built to remove. The user declares what to investigate and with what, the model decides how, and the system still guarantees that the run is reproducible and that every claim in it is checkable. An earlier post covers what Workflows does and how to call it. This one is about how that guarantee is built, and the three pieces of engineering that hold it up.

What reproducibility actually requires

Start with what the person checking the work needs. Nobody reviewing a credit note cares whether the analyst ran four searches or seven. They need to know five things:

• it searched the same sources as last quarter
• it was about the same company, not a close match
• it asked the same questions
• the answer came back in the same format
• every claim in it points to a source you can open

That is the whole list. The path the run took is not on it. The path is easy to check, so people use it as a stand-in for reproducibility, and it is also the most expensive thing to guarantee.

Fixing the path is still a fair way to get the rest of the list, and most workflow tools do it that way. You can inspect the run before it executes, it costs roughly what you expect, and it fails somewhere you can point at. Those are advantages and we took them seriously before deciding against them.

The cost is the second hop. Take a question we get constantly. Has anything in this company’s disclosure changed in a way that matters? You read the latest transcript, and what you find there decides what comes next. Management softened the guidance language, so now the 8-K and the prior two calls matter. Or they did not, but an analyst pushed three times on inventory, so now the balance sheet detail and last week’s competitor commentary do. Neither of those was predictable from the question. Both are obvious once you have read the first document, and an agent that reads before it decides will follow them without being told.

Both shapes answer the question. The difference is what they can reach. A fixed path visits the sources someone drew an edge to. An agent visits whatever this company's disclosure turns out to require.

In research the hop worth taking is usually the one nobody anticipated. Whoever read the first document is in the best position to pick the second one.

The second argument compounds over time. Every branch condition you hard-code is a judgement you made on the model’s behalf, fixed at the moment you made it. Every decision you leave to the model gets better on its own as the models do.

Leave a decision to the model and it gets better at that decision for free.

We have run this system through several model generations. The decisions we left to the model improved without us touching them. The ones we hard-coded are the ones we have since rewritten.

So the agent works out the detail of the path. That puts the whole load on the rest of the list, because the path is no longer the thing holding a run together.

What a Workflow declares

Every item on that list can be fixed up front without fixing the path, and that is what a Workflow template does.

You set the company, the sources and the questions. The agent works out the rest, and the report comes back in the same shape.

You set out what the run needs before it starts, such as a list of companies and a date range. Next quarter you fill in the same fields with new values, instead of rewriting the instructions and hoping you phrased them the same way as last time. You give it a research plan, which lists the questions this kind of analysis has to ask. You set which sources it may search, and this is the part finance users reach for first, because the run then cannot go outside the sources you accept as evidence. You also say what a finished answer has to contain, such as a summary, a table of figures and a list of risks.

What you do not declare is how many searches to run, whether to delegate, or when the work is done.

The template owns the context and the procedure. The agent owns the execution. A run is fixed in the ways that make it comparable and free in the ways that make it useful.

Our docs list everything a template can set, with worked examples, and there is a library of example templates covering what people build most.

The orchestrator is where the plan runs

For anything spanning several companies or several angles, one agent orchestrates and the rest do the work. The orchestrator has no research tools. It cannot search, read a filing, or run code. It has a planning tool and a delegation tool and nothing else. We got this wrong the first time. An orchestrator that can do the work will do the work, fill its context window with raw tool output, and lose track of the plan it was supposed to be running. Each delegation opens a subagent with a clean context window and one part of the plan.

Resolving the company comes first

Two runs are only comparable if “NVIDIA” means the same thing in both.

In finance it frequently does not. A ticker is not a company. A company is not its listed parent, or its subsidiaries, or the entity that actually issued the debt you are analysing. Names collide, tickers get reused, companies rename and merge and spin off, and the same firm appears under four different strings across a transcript, a filing, a news wire and a broker note. An agent that resolves those strings loosely will produce an answer that looks right and is about the wrong company, and there is no way to tell from the output.

One name can mean several real companies. We resolve it once, and every tool after that takes the id rather than the name.

RavenPack has been solving this since long before any of it was agentic, and the rest of the system is built on it. We resolve the company once, against the knowledge graph, and every tool then takes that resolved id rather than the string the user typed. Search, transcripts, filings, tearsheets and the calendar all take the same id.

A whole class of failure then disappears. When the agent asks for the filings of an id rather than the filings of “Volkswagen”, it cannot hand back Volkswagen Financial Services when you meant Volkswagen AG, and you can prove that the sources it returns are about the company the user asked for. You can also audit the run afterwards, because the company a claim refers to is a resolved id and not something inferred from the surrounding text.

The rest of the tools follow the same idea. Our earnings calendar returns the dates, the EPS figures and the links to the transcript and filings for each one, and it already knows the company’s fiscal calendar. The agent never has to search for a document it could have been handed, or guess whether “the latest quarter” means the fiscal one. If you know something about how research in your field works, put it in the tool result.

If you are building agents over financial data, build this part first. It is the cheapest piece of infrastructure described here and it has prevented more wrong answers than anything else we have shipped.

Grounding: every claim points at a source

Every sentence in a finished answer is tied to the document it came from. Because the path is not fixed, a run uses whatever sources it turned out to need, found as it went, sometimes across several agents. We cannot wire up attribution in advance. It has to be built while the run happens, which means what gets recorded is what the run actually did.

Nothing becomes a citation unless a tool in that run produced it. We build the registry while the run happens, so what the answer can cite is exactly what it retrieved.

We register every tool result as it comes back, along with the individual references inside it, such as the specific chunk of a transcript or the specific passage of a filing. As the model writes, it marks the text it is supporting and names the sources it used. We resolve those marks against what was registered during that run and turn them into citations while the answer is still streaming. A mark that does not resolve to something the run retrieved never becomes a citation.

So the agent cannot cite a source it did not read. The only things it can cite are results of tools that ran. What the answer claims about its own sources is checked against what the run did, rather than taken on trust.

The grounding data also stays with the conversation. A follow-up question three turns later can resolve a citation from the first turn without re-running the tool that produced it, and a user clicking a reference in an answer from last week still lands on the passage it came from.

A side effect: citations are also how the agents talk

We did not plan this next part. Once the grounding data exists while the run is going, it becomes the obvious thing to pass between agents.

Most multi-agent systems pass summaries. A subagent finishes, writes a paragraph about what it found, and hands the paragraph up. That works, but it loses the sources at the point where you need them. The orchestrator now believes something it cannot attribute. Ask it to cite that claim later and it will either invent a source or re-fetch everything the subagent already read.

So we pass sources instead of prose, in both directions. A subagent inherits the orchestrator’s sources with the content attached, so it can build on what has been found rather than search for it again. When the subagent returns, we attach its citations to the message the orchestrator receives. The orchestrator can then quote a passage it never read itself, in any later turn, and nobody has to trust a summary.

Python instead of more tools

The third piece follows from the same decision to leave judgement with the model.

Model coding ability has improved faster than any other capability we depend on. Two years ago, giving a research agent a Python interpreter meant supervising it. Now the code is routinely better than what we would have written for the same task, and it is written in the moment, for the specific question, against the specific data.

That changes what a tool is for. When an agent cannot do something, the instinct is to add a tool. But every tool is something you design, name, describe and maintain, and it takes a slot in a list that gets worse as it grows. You can also only add tools for the analyses you thought of.

So our most useful tool contains no analysis at all. The agent writes Python against a library that exposes the platform, including prices, financials, company data, search, entity lookup, holdings and charting. It can pull a price series, join it to a screen result, compute the metric nobody built a tool for, and chart the output. None of that had to be anticipated, and the tool list did not grow.

Handing an agent an interpreter only works if the interpreter can reach your data and nothing else. The two easier answers both fail. A shell on real infrastructure gives the model a way to send data out. An approval prompt on every action gives you a product nobody uses.

Everything the sandbox can reach passes through one layer. New libraries go above that layer, so the platform can grow without the security code changing.

This is what works for us. The code runs in a sandbox with no credentials the model can read and no general route out. It reaches our platform only through libraries we upload into the session. Every call those libraries make goes through one shared layer, which attaches a service key the model never sees, applies the requesting user’s own entitlements, and records what the code consumes. Adding a new capability means writing a self-contained package and one line in a registry. The new package gets all of that without its author touching any of it, so a data scientist who knows a dataset can put it in front of every agent without going near the security code.

Where this leaves us

Reproducibility turned out to rest on four things that are true about a run: what it was asked, who it was about, what it was allowed to look at, and what it can prove. Pin those four and you can leave the rest to the model. The rest is also the part that gets better on its own every time the models do.

---

About Workflows

Workflows turn a research procedure into something you can run across your whole coverage universe, with every claim tied to the document that produced it. Start with creating a template, or take one from the example templates and edit it.

---

Juan Luis Domínguez

AI Engineer