Preclinical Data Retrieval, Rebuilt Around Agents

Pharmaceutical research runs on decades of accumulated study data, but that depth is also the problem: the information is fragmented, unstructured, and often locked in legacy PDFs whose metadata has degraded through years of system migrations. Bayer’s Preclinical Information Center (PRINCE), built with Thoughtworks, is a case study in applying agentic AI to this mess — evolving from a search portal into a multi-agent system that researchers can query in natural language and even task with drafting regulatory documents.

The architecture decisions behind PRINCE are best understood through two complementary lenses. Context engineering governed what information each model received, what it deliberately did not receive, and how context flowed between specialized steps like research, reflection, and writing. Harness engineering covered the scaffolding around the models: orchestration, tool boundaries, state persistence, retries, fallbacks, validation, reflection loops, observability, and human review.

From Fragmented Repositories to a Unified Gateway

The initial problem at Bayer was familiar to anyone working with large enterprise data: information was scattered across disparate systems, traditional search relied on rigid Boolean logic that struggled with preclinical terminology, and extracting insights across documents required punishing manual effort. The result was incomplete views of compound data and researchers spending more time hunting for information than interpreting it.

PRINCE was launched to consolidate siloed structured study metadata into a single searchable interface — a useful but limited first step. Much of Bayer's authoritative knowledge resided not in that metadata but in unstructured PDF study reports, which often contained correct information despite missing or incorrect accompanying annotations. This drove the platform through three deliberate phases:

  1. Search: unified access to thousands of nonclinical study reports, powered primarily by structured metadata and advanced filters.
  2. Ask: RAG-enabled natural language question answering over unstructured data, including scanned PDFs from historical reports.
  3. Do: a multi-agent active research assistant that handles complex queries, orchestrates workflows, and supports drafting regulatory documents.

Context Engineering: Shaping What Models See

The transition from keyword search to conversational "ask" mode solved the access problem, but real-world queries are rarely satisfied by a single document lookup. A researcher may ask a compound-level question that spans multiple study types, requires cross-referencing endpoints, or demands synthesis across both structured tables and free-text narrative sections.

PRINCE's agentic architecture broke this down into specialized steps. At query time, context was routed deliberately — each agent received only the contextual slice appropriate to its function, whether that was retrieving candidate studies, reasoning about endpoints, or assembling an answer grounded in specific cited passages. This prevented the model from drowning in irrelevant context and made the provenance of each output traceable.

Harness Engineering: Keeping Agents on the Rails

Agentic systems are only as reliable as the control plane around them. PRINCE's harness encompasses several layers. Orchestration determined how subtasks were sequenced and how intermediate results were passed between agents. Tool boundaries were explicitly defined so that models could not wander outside their intended capabilities. State persistence ensured that multi-turn research sessions survived interruptions and could be revisited.

Given the stakes in pharmaceutical research, the harness had to include recovery mechanisms — retries on transient failures, fallbacks when an agent returned unusable output, and validation checks on structured query generation. Reflection loops allowed the system to critique its own intermediate answers before final presentation. Observability tracked every step so engineers could diagnose failures rather than guess at them.

Trust Through Architecture, Not Assurance

The most critical engineering choice was embedding human review into the workflow rather than treating automation as a replacement for scientific judgment. PRINCE's outputs — particularly draft regulatory documents — are presented with transparent citations that point back to the exact source passages in the underlying study reports. This makes the system's reasoning auditable and gives researchers a clear path from an answered question back to the authoritative gold-standard information.

That deliberate combination of context and harness engineering is what separates a working agentic system from a promising demo: the models handle the complex cognitive work of synthesis, while the surrounding software guarantees that the results are grounded, recoverable, and reviewable.

Engineering the Agentic RAG Backend

The platform is built as an interactive conversational interface (React) on top of a FastAPI service, with a LangGraph orchestration layer coordinating the multi-agent workflow. The backend routes each request through a staged pipeline: intent clarification, planning, research (via both RAG and Text-to-SQL), data validation, and final synthesis by a Writer agent. Deliberate pause points and feedback loops are embedded in the graph to verify data completeness before the workflow advances.

Distributed Data and State Persistence

Researcher agents pull from a distributed data ecosystem rather than a single store. Vector embeddings of study reports live in OpenSearch and serve as the retrieval knowledge base. Curated structured data, produced by ETL and harmonization pipelines, is queried through Athena. Execution state is tracked rigorously: after each LangGraph node executes, the checkpointer persists that state to PostgreSQL, while broader application-level state is maintained in DynamoDB.

Model Access and Fallback Strategy

All LLM access goes through internal GenAI platforms that host models from OpenAI, Anthropic, Google, and open-source providers. These platforms expose a unified OpenAI-compatible endpoint, which simplifies swapping models per task and centralizes control-plane concerns like rate limiting.

Resilience is treated as a first-class design requirement. The system handles model failure with automatic retries before falling back to an alternative model or platform. Retries are layered at two levels: the individual LLM call and the logical node (an entire step in the agent's plan). When an error persists, agents are fed the error context so they can alter their trajectory or devise a new plan rather than blindly retrying the same failed path.

Observability and Evaluation Pipelines

Production monitoring uses CloudWatch for system health, with Langfuse as the primary tracing and debugging tool. Langfuse captures detailed traces of all traffic and also stores evaluation datasets, linking performance scores directly to specific failures. Evaluations run on two schedules: live traffic is assessed daily using the RAGAS framework, while dataset evaluations are triggered only after significant changes to the core workflow, prompts, or models.

Context Discipline as a Design Rule

A core architectural principle is that larger context windows do not justify dumping all available information into a single prompt. Early versions that did so proved harder to steer and harder to evaluate. The system (internally referred to as PRINCE) avoids this by segmenting context by stage: planning context for the Think & Plan agent, retrieval context for the Researcher, evidence context for the Reflection agent, and synthesis context for the Writer. This separation reduces context pollution, making each stage easier to debug and improve independently.

Orchestrating research with specialized agents

PRINCE’s research workflow is built on LangGraph and splits responsibilities across three agents — Researcher, Writer, and Reflection — to tackle requests that need reasoning and multiple tool calls. Clarifying user intent happens up front, before any retrieval begins.

Clarify intent before retrieval

As PRINCE expanded into domains like toxicology and pharmacology, queries became harder to route automatically. Instead of running costly trial-and-error across every available source, the system asks clarifying questions first to pin down the relevant domain or data type. A planned domain-level selection in the UI will let users pre-filter valid tools, and the system can already offer AI-assisted source recommendations: when no source is selected, or several are chosen without focus, the model analyzes query intent and suggests sources, which the user can accept, adjust, or reject.

This is a deliberate “fail-fast” step. It keeps ambiguous requests from consuming execution resources and, from a context engineering view, makes the first workflow decision before any retrieval starts, constraining which tools, domains, and data sources are in scope for the agents that follow.

Think & Plan as process reflection

The Think & Plan step gives the system space to reason about next steps before acting, in the style of Anthropic’s Think tool. Its role is process reflection: evaluating whether the agent is on the right trajectory and making proper progress, rather than judging the data itself.

That distinction matters in long multi-step runs. With 50 sequential actions, the system must ask at each point whether the steps are sequenced correctly and whether the trajectory still serves the user’s goal. The thinking step provides that metacognitive check.

It also solved a specific problem as PRINCE grew. Early versions had two tools — one for RAG retrieval, one for Text-to-SQL. Adding data sources multiplied available tools, and similar domains began to overlap: structured metadata versus unstructured reports, study summaries versus detailed experimental data. The LLM often picked a plausible but wrong tool when several were close in function. Giving the model a dedicated reasoning step before tool selection produced a noticeable improvement in accuracy.

The same step supports orchestration across dependent calls. A typical query might first hit structured metadata to find relevant study IDs, then use those IDs to pull details from unstructured reports, then synthesize across both. Without explicit process reflection, the system would run those steps without checking that each one is getting closer to the goal.

Researcher Agent: structured and unstructured retrieval

The Researcher Agent gathers information across scientific domains. Data consistently divides into structured and unstructured categories, and although domain implementations differ — for example, Snowflake Cortex Analyst handles pharmacology Text-to-SQL, while toxicology uses custom methods — the retrieval fundamentals stay consistent.

Managing a single agent with a flat tool list has become harder as domains grew. Terms like “studies,” “findings,” and “assays” recur across domains but point to different datasets, schemas, and regulatory meanings. “The study” might mean a repeat-dose toxicology run, a cardiovascular safety pharmacology package, or one assay in an aggregated table.

The architecture is evolving toward domain-specific sub-agents, each owning its toolset and prompt instructions that encode the domain data model and authoritative sources. The top-level Researcher would act as a coordinator, routing a query to the right sub-agent rather than knowing every tool itself. Users still see one Researcher; internally, each domain can evolve its retrieval recipes without destabilizing the rest.

Retrieval itself uses two patterns:

  • RAG for unstructured data, chiefly PDF reports.
  • Text-to-SQL for structured data in Amazon Athena.

RAG for unstructured data

The corpus spans thousands of preclinical study reports, many scanned, with complex tables and decades of history. Ingestion centralizes PDFs into an S3 data lake, runs them through an extraction pipeline, and normalizes the text into structured JSON. Chunking preserves scientific context at retrievable sizes. Each chunk is tagged with study- and section-level metadata from Amazon Athena — study ID, compound, species, route, page, parent section — enabling later filtering. These annotated chunks are embedded and indexed in Amazon OpenSearch Service, which also receives daily deltas for new or updated reports.

Query time runs a multi-stage pipeline. For the example “Were any of the following clinical findings observed in study T123456-2: piloerection, ataxia, eyes partially closed, and loose faeces?”:

  1. Keyword extraction. An LLM pulls terms suited to keyword search, such as piloerection, ataxia, eyes partially closed, and loose faeces.
  2. Metadata filter generation. A second LLM output produces a filter like eq(study_id, T123456-2), built with few-shot prompting covering diverse filter permutations.
  3. Query expansion. A smaller, faster model generates n=5 semantically similar queries to catch phrasing variations. For the example, variants paraphrase “goosebumps,” “lack of coordination,” and “watery stools.”
  4. Hybrid retrieval. Each expanded query runs against the OpenSearch index in parallel, applying the metadata filter first to cut the candidate space from millions of vectors to tens or hundreds, then combining semantic kNN and keyword search.
  5. Weighted scoring. Within each hybrid search, semantic results are weighted 0.7 and keyword results 0.3, a balance chosen experimentally for this data.
  6. Aggregation. Results from all five searches are merged, keeping unique chunks and their highest weighted score for an initial ranking — retrieving roughly k=~20 candidates.
  7. Reranking. A cross-encoder model (bge-reranker-large) scores each candidate against the original question and keeps the top k=7 chunks as final context.
  8. Generation with citations. A reasoning model produces the answer from those seven chunks and automatically links claims back to source chunks.

The entire pipeline is monitored with Langfuse for observability, performance, and quality.

Text-to-SQL for structured data

Queries like “Give me 50 example studies done on RAT” or specific numerical assay results need precise filtering and aggregation better handled in SQL. The Researcher Agent routes those to the Text-to-SQL tool, which converts natural language into executable Athena queries through several steps:

  • Intent recognition. The query is analyzed to identify requested data points and filters.
  • Schema selection. Rather than emitting the full database schema, only the components relevant to the query are injected into the LLM context.
  • Dynamic few-shot prompting. Hand-picked examples of complex queries and their correct Athena SQL are stored in a vector collection as a “semantic layer.” Vector search retrieves examples similar to the incoming query and adds them as in-context demonstrations. The example set grows as new challenges appear.
  • Generation and validation. A code-capable model produces the SQL. Essential columns, like study_id and study_title, are always included so downstream synthesis can identify rows. Validation enforces SELECT-only operations — DELETE, INSERT, and UPDATE are blocked. An earlier LLM review step was removed after testing showed it incorrectly rejected valid queries more often than it caught errors.
  • Execution with limits. Queries run against Athena and fetch at most 50 records to control response size.
  • Error handling. On failure, the error message, generated query, and original context return to the same model for correction. The generate-and-execute cycle retries up to three times before reporting failure.

Reflection Agent: data validation

Where Think & Plan checks the process, the Reflection Agent checks the data. It compares retrieved context against the original query and identifies gaps or missing information. If evidence is insufficient, the agent generates follow-up questions. Those questions return to Think & Plan, which triggers further retrieval and repeats the loop until the evidence satisfies the Reflection Agent — or the workflow proceeds to answer synthesis.

This separates two failure modes. A system can execute valid workflow steps and still retrieve too little data; it can also have sufficient data while making poor progress. The two agents handle those cases independently.

Writer Agent: grounded synthesis

The Writer Agent converts retrieved evidence into the final answer. It does not discover new information; it synthesizes supplied context while obeying user and domain formatting constraints. Every claim must be grounded in the retrieved chunks and carry citations back to source study IDs, since verifiability is essential in a regulated setting.

Complex outputs, like multi-section summaries or partially filled regulatory templates, can run through an internal review loop: the Writer drafts, a reviewing step checks for missing sections, inconsistent tables, or gaps against the original question, then sends targeted revision instructions back to the Writer. This draft-level reflection complements the Reflection Agent’s data-sufficiency focus. All such outputs remain drafts for expert review; qualified personnel author and approve final submissions.

PRINCE therefore has three complementary reflection loops:

  • Process reflection (Think & Plan) catches bad trajectory, wrong tool choice, or poor sequencing.
  • Data reflection (Reflection Agent) catches thin evidence, missing context, or coverage gaps.
  • Draft reflection (Writer review) catches missing sections, incomplete tables, or synthesis gaps.

Routing context by role

The workflow does not rely on growing prompt size. It routes the right context to the right capability: planning context to Think & Plan, retrieval context to the Researcher, evidence context to the Reflection Agent, and synthesis context to the Writer. Concretely, Text-to-SQL injects only schema relevant to the query, not the full database schema; the Reflection Agent sees the original question alongside collected evidence, not the full run history; and the Writer receives curated chunks with citation constraints, not raw retrieval output. Splitting one monolithic agent into this workflow means each piece can be evaluated, debugged, and improved in isolation.

Trust Mechanisms in Production

For a system like PRINCE operating in preclinical drug discovery, user trust rests on more than answer accuracy. It requires transparency into the system's process and the ability to verify outputs against source material. PRINCE integrates several features to support this.

Transparency and Verification

Because the agentic workflow is iterative and can take time to produce a final answer, PRINCE displays the intermediate steps it executes—including formulated queries and tools used—so users can follow the system's reasoning process. When relevant context chunks are identified, links to the source materials appear on screen, showing users exactly which information was shortlisted for the response.

Citation-based factuality verification is a core component. Every generated answer is accompanied by citations referencing the original documents and structured metadata. Users can hover over any sentence to see the corresponding citation, which includes a link to the source document, the page number, and the exact quote supporting that part of the answer. This granular level of detail simplifies the human review process and strengthens the credibility of the output.

Evaluation Strategy

PRINCE uses a two-tiered evaluation approach to assess performance and reliability.

  • Dataset Evaluations: Run whenever the core workflow, prompts, or underlying models change significantly. These use curated datasets with pre-defined reference answers prepared by subject matter experts and stored in Langfuse. A custom evaluation script compares each generated response against the reference answer, producing quantitative metrics including Faithfulness, Answer Relevancy, Context Relevancy, Answer Accuracy, and Semantic Similarity with Reference. Given the agentic design, evaluating metrics at different workflow stages—analogous to a testing pyramid—is as important as evaluating end-to-end performance.
  • Live Traffic Evaluations: Performed daily as a batch job on real user queries from the live environment, without pre-defined reference answers. Metrics like Faithfulness and Answer Relevancy are still assessable. These evaluations are essential for monitoring system behavior, detecting potential hallucinations in production, and understanding performance on diverse real-world queries.

Continuous Monitoring

Using platforms such as Langfuse, the team continuously monitors PRINCE's outputs to identify potential biases, errors, or areas for improvement. This ongoing surveillance ensures the reliability and safety of responses in the production environment.

Error Handling and Recovery

The complexity of PRINCE's multi-step workflow demands robust error handling so the system can recover gracefully from failures at any stage without restarting the entire process. The approach includes several complementary mechanisms.

  • State Persistence: The state of the entire workflow graph is stored persistently, allowing execution to resume directly from the failed node. The Agent State, representing each agent's progress through the workflow, is stored in Postgres. Other application state—logs, intermediate steps, and citations—is stored in DynamoDB. This separation is critical for robustness in a stateful agentic system.
  • Built-in Retries: The system is configured to automatically re-execute a step a pre-defined number of times upon transient failure before signaling a more permanent error.
  • User-Initiated Retries: Users can manually retry a failed query through the interface. The system uses the persisted state to continue from the failure point, skipping steps already completed successfully in the prior attempt. This saves computational resources and improves user experience.
  • Framework-Level Support: LangGraph, the underlying framework, provides built-in capabilities for managing workflow state and handling errors within the graph, offering a solid foundation for resilient agentic workflows.
  • LLM Fallbacks: Custom fallback handling has been implemented. If a call to a primary LLM provider or model fails after several retries, the system automatically switches to an alternative LLM from a different provider. This maintains availability and responsiveness even when external model platforms experience downtime.

This comprehensive approach minimizes the impact of transient failures, reduces the need for users to restart complex queries from scratch, and avoids redundant execution of successful steps and LLM calls—contributing to cost and latency savings essential for production readiness.

These mechanisms represent harness engineering in practice. The LangGraph workflow acts as the control layer around the agents, defining which components can act, which tools they can use, where the workflow can pause, how failures are retried, how state is persisted, and when the system transitions from research to reflection to writing. This harness makes the system less opaque and more reliable than an unconstrained autonomous agent, providing clear control points for recovery, inspection, evaluation, and human intervention.

Improving Metadata with NER

The accuracy of structured metadata in Amazon Athena directly impacts the Text-to-SQL component's performance and data discoverability. Historical data migrations and varied annotation practices across Bayer's laboratories over the years have left some metadata incomplete, missing, or incorrect.

To address this, the team has developed a utility system that uses Named Entity Recognition (NER) to extract and create accurate annotations directly from study PDFs. The process reads the textual content of preclinical reports and identifies key entities such as study IDs, compound names, species, routes of administration, dosage information, and clinical findings, then generates structured annotations based on those entities and their relationships.

The team is actively integrating this utility into data pipelines to automatically correct and enrich Amazon Athena data. An evaluation system is being developed that provides a confidence score for each extracted field. High-confidence annotations will automatically update the corresponding database entries; lower-confidence fields will be quarantined and flagged for human review. This hybrid approach aims to continuously improve metadata quality while preserving data accuracy.

Iterative Development in Practice

PRINCE has been available to end-users since early 2024, with the agentic integration introduced later that year. Real-world feedback has driven iterative development throughout. The guiding principle is not to wait for absolute perfection before seeking user feedback, but to deliver value early and refine continuously based on actual usage.

Initially, development focused squarely on achieving desired accuracy and performance for core functionalities, even at higher cost. The team recognized that premature cost optimization could compromise effectiveness and hinder adoption. Only after reaching the target accuracy and performance levels did attention shift to cost optimization, ensuring efficiency gains did not degrade user experience or result quality.

User feedback, monitoring data, and insights from expert scientists are continuously fed back into the development cycle, driving refinements in architecture, retrieval techniques, agent behaviors, and the user interface—ultimately enhancing performance, usability, and scientific impact.

Engineering the Whole System, Not Just the Prompt

The PRINCE deployment at Bayer makes a clear case that production-ready agentic AI in a regulated enterprise setting depends less on model choice and more on disciplined engineering of both context and harness. Context engineering ensured each model received the right information, and only that information, at each workflow stage. Harness engineering kept the overall process bounded, observable, and recoverable.

Context and Harness: The Two Levers

Getting value from vast preclinical data repositories required combining robust data infrastructure with retrieval methods such as RAG and Text-to-SQL, orchestrated through a multi-agent system. But the decisive factor in making that system trustworthy was reliability engineering applied across several layers:

  • Error handling and fallbacks: The system was designed to degrade gracefully when individual components fail.
  • State persistence: Workflow state was persisted so that long-running agentic processes could be resumed rather than restarted.
  • Transparency and explainability: Granular citations and step-by-step reasoning visibility were built in to establish user confidence.
  • Continuous evaluation: The system's outputs were monitored against expected performance metrics over time.

Why Explicit Control Still Matters

As base models improve, some of today's harness components may become thinner or migrate into native model capabilities. However, in research environments where traceability and reviewability are non-negotiable, explicit control over context, workflow state, recovery, and verification remains a necessity rather than a stopgap.

The results so far show meaningful gains in data accessibility and research efficiency at Bayer, shifting how scientists query and interact with preclinical information. The work represents a step toward more capable research assistants, but the engineering principles applied — particularly around context curation and workflow containment — are broadly applicable to any organization aiming to run agentic systems in a regulated domain.

Acknowledgments

The author acknowledges the Bayer team's Adam Zalewski, Annika Kreuchwig, Carlos Henrique Vieira-Vieira, Jobst Löffler, and Jonas Münch, as well as the Thoughtworks team: Bala Hari, Balu Saravanan, Bernice Mercy Sharon M, Deril X, Jigar Jani, Manibalan Baskaran, Nafis Aslam, Priyalakshmi R, Rohit Bansal, Sai Prabhanj Turaga, Saksham Srivastava, Shivam Sehgal, Sowmya Adimoulame, and Subhashini Rajamani.

AI assistance was used during article preparation for brainstorming, outlining, and language review.

Disclaimer

All activities described conform to Bayer's information classification, data governance, and external communication policies. Nothing herein constitutes a claim regarding regulatory decision-making or product performance.