RAG: Key Aspects of Performance: Metrics and Measurement

Evaluating Retrieval-Augmented Generation (RAG) pipelines ensures these systems are accurate, reliable, and effective. RAG pipelines combine retrieval mechanisms with generative models to generate contextually relevant and precise responses. As I am measuring RAG vs Knowledge Graph enabled RAG performance, I am considering the below metrics. I would welcome all the experts' inputs, missing aspects, and metrics!

Key Dimensions and Metrics for RAG Performance

The Retrieval and Generated related performance aspects or dimensions is discussed below:

RAG Performance Aspects with examples by author

RAG Performance Metrics and Measurement Process

Below detailed are the top Metrics. Related measurement process is explained below with examples:

1. Accuracy

Measures the overall correctness of the responses generated by comparing the correct answers to the total number of questions. Attributes to the “Fairness” aspect above.

Ex: In a medical RAG system, suppose there are 100 patient queries about symptoms. If 90 of these queries are answered correctly based on medical guidelines and evidence, then the accuracy is calculated as:

A user asks, “What are the symptoms of diabetes?” The system should provide accurate symptoms such as increased thirst, frequent urination, extreme fatigue, and blurred vision. If the response includes these symptoms and matches medical guidelines, it is considered accurate.

2. Cosine Similarity

Measures the similarity between the retrieved context and the ground-truth answer using vector representations. Measures the Context Relevance aspect of the RAG. Similarity between the retrieved context and the ground-truth answer using vector representations.

For a technical RAG system, if a user asks for troubleshooting steps for a specific software issue, the system’s retrieved documents are compared with an ideal answer. A high cosine similarity score indicates high…