Leading Insurer Selects the Best LLM for Underwriting with Skan

Skan’s Process Intelligence Platform allowed the carrier to conduct A/B testing of different LLMs to identify the optimal model for their use case. In addition, Skan showcased how these outputs differed by case type and complexity, illuminating the entire end-to-end process.

Phase 0: Established current underwriting process baseline and metrics by installing Skan on systems used by the entire population of underwriters.

Phase 1: Designed an experiment to measure the impact of the LLMs by splitting the participants into sub-groups and having each sub-group perform queries.

Phase 2: Assessed each LLM using Skan to measure metrics, such as Processing time, and Skan’s Clickstreams, a playback of the entire end-to-end case journey, to validate query output accuracy.

Phase 3: SMEs reviewed all outputs, provided quality ratings, and along with Skan’s process metrics, assessed each LLM. 

By A/B testing, Skan was able to help the client establish a baseline for underwriting operations and then test various LLMs to see how they affected outputs and decisions. Skan help them determine the optimal LLMs for specific use cases, reducing search query time by 55% over traditional search functionality.

In some instances, processing time was reduced significantly due to the use of LLMs, and Skan measured improvements down to the minute. The client also created LLM ‘prompt’ training material so that operators use LLMs for complex queries rather than simple queries for which answers are readily available.

Skan enabled the client to quantify their business case for using LLMs and to select the best LLM for their organization.