# Evaluating Doximity GPT with Ground Truths and LLM Performance Metrics

DevFeed: [Evaluating Doximity GPT with Ground Truths and LLM Performance Metrics](<https://devfeed.tech/articles/beyond-accuracy-20023.md>)

Original publisher: [Read original article](<https://technology.doximity.com/articles/beyond-accuracy>)

Author: Doximity

Published: 2024-05-10T13:00:00Z

Content type: article

Language: en

Sources: [Doximity](<https://devfeed.tech/sources/doximity.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>)

Tags: [evaluation](<https://devfeed.tech/tags/evaluation.md>), [hipaa](<https://devfeed.tech/tags/hipaa.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

## AI overview

This article explains how Doximity evaluates its Doximity GPT medical writing assistant. It discusses using ground truths and golden datasets to establish baseline metrics, comparing contender models, and combining standard metrics, manual review, benchmarks, and LLM-assisted evaluation to assess relevance, coherence, factual accuracy, and ethical compliance.

## Source excerpt

At Doximity, we go to great lengths to ensure the quality of our products aligns with the standards physicians require. Across various industries, Large Language Models (LLMs) have become the backbone of numerous applications, driving advancements in everything from natural language processing to automated content creation. As we continue to develop products that make use of these LLMs, the need for rigorous and comprehensive evaluation of their outputs has never been more critical. Strap in as we explore the process for evaluating our Doximity GPT product, Doximity's HIPAA-compliant medical writing assistant, focusing on the importance of using "ground truths" to establish baseline metrics and the relative performance of contender models. A Brief Overview of How LLMs Generate Outputs LLMs are trained on vast amounts of textual data in order to learn patterns, structures, and nuances of language. By processing this data, these models develop the ability to generate text that mimics human writing. The output generation process involves the model understanding the input prompt, enhancing its focus via a system prompt, running all of that through its learned information and constructing a coherent and contextually relevant response. While this capability makes LLMs incredibly versatile, it also introduces unique challenges in ensuring the outputs meet specific quality and accuracy standards. LLM Evaluation: What is it and Why Does it Matter? Accuracy, precision, and recall are standard metrics that help evaluate different aspects of classification model quality in machine learning, and these metrics are still incredibly important for LLM evaluation as well. Additional traditional methods include manual review and comparison against predefined benchmarks. However, as LLMs grow more complex, these methods become less feasible due to the sheer volume of outputs and the nuanced understanding required to assess them. In addition to standard machine learning performance metr