-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
adding context relevance metric to RAGAAF
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
- Loading branch information
Showing
1 changed file
with
12 additions
and
0 deletions.
There are no files selected for viewing
12 changes: 12 additions & 0 deletions
12
evals/metrics/ragaaf/prompt_templates/context_relevance.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Copyright (C) 2024 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
class ContextRelevance: | ||
name = "context_relevance" | ||
required_columns = ["question", "context"] | ||
template = """- Context Relevance: Context Relevance measures how well the context relates to the question. | ||
- Score 1: The context doesn't mention anything about the question or is completely irrelevant to the question. | ||
- Score 2: The context only identifies the domain (e.g. cnvrg) mentioned in the question and provides information from the correct domain. But, the context does not address the question itself and the point of the question is completely missed by it. | ||
- Score 3: The context correctly identifies the domain and essence of the question but the details in the context are not relevant to the focus of the question. | ||
- Score 4: The context correctly identifies domain mentioned the question and essence of the question as well as stays consistent with both of them. But there is some part of the context that is not relevant to the question or it's topic or it's essence. This irrelevant part is damaging the overall relevance of the context. | ||
- Score 5: The context is completely relevant to the question and the details do not deviate from the essence of the question. There are no parts of the context that are irrelevant or unnecessary for the given question.""" |