Foundry Labs / From the community
AI for Insurance Claims: Design the Review Around the Evidence
In this article
The Model Is Not the Hard Part
When an insurer adopts an AI system for damage assessment, the most consequential engineering decisions are rarely about which model to use. They are about how to verify that the model, and every component around it, is doing what the business actually needs, at scale, across many clients and regional variants, and without introducing errors that undermine adjuster trust.
At Tokyo AI's AMA #11 on 5 August 2025, engineers and researchers from Tractable presented a grounded, reproducible framework for exactly this problem. The session anchored on the talk "From Lab to Live: AI System Evaluation in Practice" by Filippos (Fil) Siozopoulos, Senior Applied Researcher at Tractable. The announced programme also covered Japan market entry, multi-agent platforms for insurance workflows, and property-damage pipelines.
This article examines how those evaluation choices can inform an insurance claims-review workflow.
Event Details
| Event | Tokyo AI AMA #11: AI x Insurance |
| Date | 5 August 2025, 18:00-21:00 JST |
| Location | Minato City, Tokyo |
| Organiser | Tokyo AI (TAI), supported by GHOVC and Tractable |
| Format | In-person, four talks and networking |
| Community | Tokyo AI (TAI) |
Why Evaluation Is the Central Problem
Early AI systems for damage assessment were often monolithic: one model handling many tasks, with retraining required whenever a new task or client was added. Fil describes this architecture as producing slow iterations, which was tolerable when each deployment had a dedicated owner and evaluation could be handled ad hoc. As soon as the system needs to serve multiple clients and regional configurations simultaneously, that model breaks down.
The move toward component-wise design, where each model is highly specialised on a single task and the output of one model becomes the input of the next, resolves the iteration problem but creates a new one. An update to any component affects the performance of everything downstream. Without a robust evaluation framework, teams cannot make confident go/no-go decisions. They are either deploying changes they cannot fully verify, or they are moving so slowly that the system cannot improve.
Fil frames the entire evaluation challenge around three guiding principles:
- Rapid correctability: detect errors at their onset and act on fixes immediately.
- Scalability: the evaluation system itself must scale as the number of clients grows.
- Efficient data collection: a reusable pipeline to ingest production data, with robust storage for on-demand retrieval.
These are engineering constraints, not quality ideals. They determine whether iteration is practically possible.
Evaluation Scope: Component, End-to-End, and Profiles
Fil organises evaluation across three questions:
- Scope: what is being evaluated, a single model component, the system end-to-end, or a profile?
- Methodology: what evaluation algorithm and what data?
- Strategy: how is evaluation applied in practice, and what tooling does it require?
The approach Fil described runs rigorous component-wise testing in the lab before graduating to end-to-end evaluation. The added layer is the concept of profiles: different clients or regions often require different system variants. A component upgrade that improves performance for one profile may degrade another. The evaluation strategy must account for this explicitly. Fil notes that when covering different use cases, an update may be an upgrade for some but a downgrade for others, and the system should make it possible to apply a component change to one profile but not others, as results support.
This is a structural requirement, not a nice-to-have. Teams that evaluate a single global model and deploy uniformly are systematically blind to differential impact across their client base.
Data Strategy: Production First, Manual Labelling on Demand
The most common mistake in evaluation data strategy is over-relying on manually labelled test sets collected before deployment. Fil's recommendation is direct: use production data for general evaluation, and reach for manual labelling only when targeted analysis of a specific error mode requires it.
Production data has two decisive advantages: volume and automatic collection. It captures the actual distribution the system encounters. Its limitations, noisiness and the fact that it can reflect biases introduced by the current production system, are manageable when the team understands them. Manually labelled data offers fine-grained control and, at best, high alignment with client standards, but the cost and effort are high, and external labellers may introduce misalignment with how the client actually defines a correct assessment.
On volume, Fil recommends running a statistical test simulation rather than picking a number intuitively. The inputs are expected performance levels and the sensitivity required to detect a meaningful change. The right statistical tests to consider include the Paired Z-test and McNemar's test. The example in the deck makes the point clearly: a shift from 80% to 82% accuracy may not be statistically significant for a given dataset size, while its larger change is marked significant. Those are illustrative results, not a universal significance rule: sample size and the test assumptions matter. Dataset volume is use-case dependent; there is no universal answer.
On sampling, Fil recommends doing both: random sampling from production to capture the expected distribution, and targeted sampling to evaluate specific error modes. Random sampling is simple and represents real conditions; targeted sampling allows focused investigation but will not reflect production frequency.
Trust-Eroding Errors: The Metric Blind Spot
The most practically important concept in Fil's session is what he calls trust-eroding errors. These are rare errors that may substantially damage user trust when they appear in production, even when aggregate accuracy metrics look healthy.
The reason is perceptual. A claims reviewer who encounters a serious, visible mistake by the AI system, even once, will update their mental model of the system's reliability in ways that a percentage-point metric cannot capture. Human perception does not always align with the numbers. Five correct outputs followed by one badly wrong output may feel like a system that does not perform well, even if the aggregate accuracy is 83%.
Fil's takeaway is precise: carefully assess what trust erosion means for your specific product, in collaboration with subject matter experts and, where possible, with clients directly. Then ensure your evaluation framework includes explicit checks on those cases. They should not be handled only by random sampling, because they are rare by definition and will be underrepresented. They require targeted sampling, clear labelling, and dedicated evaluation passes.
This principle applies directly beyond insurance. Any AI system where a single high-visibility error can change user behaviour, whether in financial review, clinical decision support, legal analysis, or field operations, needs an explicit trust-erosion audit in its evaluation framework.
Metrics That Clients Actually Care About
Fil makes a careful point about metrics: what is measured in model training is not always what the client cares about. In Tractable's domain of repair labour hours, the regression metric that an ML team naturally reaches for (mean absolute error) may not be the most meaningful measure for an adjuster or operations manager. The deck presents MAE of 2.45 alongside a crafted metric: correctly predicting within 0.5 hours, 90% of the time. The second formulation communicates something an operations team can act on; the first does not, without further context.
The recommendation is to work with subject matter experts and clients to identify domain-appropriate metrics early, rather than defaulting to whatever the training framework produces. This is not a compromise of rigour. It is a condition for the evaluation framework to remain aligned with the business decision the AI is actually supporting.
Note that the specific figures cited here, MAE of 2.45 and 90% within 0.5 hours, are drawn directly from Fil's slide deck as illustrative of the domain. The dataset, time period, and client scope are not specified in the available material.
On-Demand and Automated Evaluation Together
Fil's final structural recommendation is that both on-demand and automated evaluation must be in place. Neither alone is sufficient.
On-demand evaluation should be easy to adopt, with capacity for heavier specialised analysis when required. A command-line tool that a researcher can install and run on their workspace is the example given. It should be possible to grow this capability organically as requirements emerge.
Automated evaluation should attach to the CI/CD pipeline and focus on a small number of key metrics. It needs careful cost design. The example is an automated pipeline triggered when a pull request is issued. Crucially, the tooling built for on-demand workloads can generally be adapted for automation, which means investing in the on-demand layer first is not wasted effort.
For tooling, Tractable uses Weights and Biases for on-demand evaluation. The workflow takes an evaluation dataset and the AI system's inference results, produces performance metrics, uploads results as a W&B Run, and generates a report for stakeholders. Fil notes that Streamlit is another portable option. The key selection criteria are ease of use, shareability, and decoupling evaluation from inference as much as possible.
The three key takeaways from Fil's session:
- Good data is the foundation. Test sets must represent what the team actually wants to measure.
- Robust and reusable evaluation tooling is worth the investment. It is how consistency is maintained as clients and variants multiply.
- Subject matter experts and clients are the best source for understanding what matters when evaluating. Build that channel early.
A Broader Insurance AI Picture
The public programme placed evaluation alongside three other perspectives. Venkat Sathyamurthy's announced session concerned Tractable's entry into Japan and the importance of local workflows and partnerships. Toru Tanaka's abstract described multi-agent platforms for insurance and financial services. Elham Asadi's session was advertised around image-quality verification, damage detection and cost estimation in a property-damage pipeline. These programme descriptions provide context for the engineering problem; the evaluation detail in this article is grounded in Fil Siozopoulos's materials.
What This Means for Enterprise AI Teams
Foundry Labs analysis: consider an illustrative claims-review workflow in which an adjuster receives customer photographs and an AI-assisted damage estimate. This is a design example, not a claim about a deployed client system.
The reviewer needs to see the original images, the proposed assessment and a clear route for handling exceptions. An image-quality component might request another photograph before a damage estimate is generated. A case that falls outside an agreed operating range could be routed for specialist review. Those decisions create evaluation requirements for the whole workflow: an apparently better damage model could still make the process worse if it increases unnecessary referrals or suppresses useful evidence.
Fil's distinction between component and end-to-end testing gives a way to investigate that problem. The team can test the changed component on its own, then measure its effect on the complete review path for each relevant client or regional configuration. Representative production samples help show ordinary behaviour; targeted cases test the mistakes that reviewers consider especially damaging. Domain experts help define both the exception categories and the operational measures that matter.
A practical prototype should therefore make its inputs, outputs and reviewer decisions inspectable. It should also preserve enough evaluation data to compare a proposed update against the existing system. Expansion to another client or workflow should depend on evidence from that configuration, with responsibility for accepting an update clearly assigned. These are concrete ways to apply the session's evaluation principles without treating a single aggregate score as a deployment decision.
Foundry Labs' enterprise AI services support teams designing and implementing production AI systems. The useful connection here is the work between a promising prototype and an operational process: data collection, evaluation, exception handling and the people responsible for each decision.
About Tokyo AI and Foundry Labs
Tokyo AI (TAI) brings engineers, researchers and other AI practitioners together for technical events. Foundry Labs provides enterprise AI engineering services in Japan. The community page has more information about TAI.
Sources
- TAI AMA #11: AI x Insurance, Luma event page
- Tokyo AI (TAI) community
- Filippos Siozopoulos, LinkedIn
- Elham Asadi, LinkedIn
- Venkat Sathyamurthy, LinkedIn
- Toru Tanaka, LinkedIn
- Foundry Labs
- Foundry Labs Tokyo AI community page
Talk materials
The following talk materials are available on request. Use our contact form and include the event title and speaker name.
- From Lab to Live: AI System Evaluation in Practice (Filippos Siozopoulos, Tractable)
About the speakers
Roles and biographies reflect the event programme.
Venkat Sathyamurthy
Venkat Sathyamurthy was introduced in the event programme as CEO of Tractable, a company developing computer-vision AI for insurance and automotive workflows. His biography describes over 25 years of enterprise technology experience, including roles at Microsoft, Adobe and DocuSign. His announced session concerned Tractable's entry into Japan and the importance of local workflows and partnerships.
Toru Tanaka
Toru Tanaka was introduced as CEO of MILIZE, Inc., a company developing AI and financial-technology solutions. His event biography describes entrepreneurial experience across financial technology and food services. The announced talk focused on LLM-driven multi-agent systems for insurance and financial workflows.
Filippos Siozopoulos
Filippos (Fil) Siozopoulos was introduced as a Senior Applied Researcher at Tractable, working across NLP, deep learning and computer vision. His event biography lists prior work at Samsung Research UK, an MSc in Machine Learning from UCL and a master's in Electrical and Computer Engineering from the National Technical University of Athens. His presentation materials describe Tractable's framework for evaluating production AI systems.
Elham Asadi
Elham (Ellie) Asadi was introduced as a Senior Applied Scientist on Tractable's Property team. Her event biography describes more than 15 years of machine-learning and computer-vision experience, including autonomous-vehicle AI at HORIBA MIRA, a PhD in Aerospace Engineering from the University of Bristol and postdoctoral research at the University of Cambridge. Her announced session concerned AI pipelines for property damage assessment.
All articles