Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ragaaf (RAG assessment annotation free) #157

Merged
merged 28 commits into from
Oct 15, 2024
Merged

Ragaaf (RAG assessment annotation free) #157

merged 28 commits into from
Oct 15, 2024

Conversation

adkakne
Copy link
Contributor

@adkakne adkakne commented Oct 11, 2024 •
edited
Loading

Description

Adding annotation free RAG evaluation feature using LLM-as-a-judge methodology.

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

  • New feature (non-breaking change which adds new functionality)

Dependencies

jsonlines, openai

Tests

Added ./tests/test_ragaaf.py for testing.

adkakne and others added 21 commits September 20, 2024 03:27
@adkakne
minimized required fields/columns in user data
669997a 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@daisy-ycguo @adkakne
add bench-target as the prefix of output folder (opea-project#133)
80e2160 
Signed-off-by: Yingchun Guo <yingchun.guo@intel.com>
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@lkk12014402 @adkakne
remove examples. (opea-project#135)
eb98d2e 
Co-authored-by: root <root@idc708073.jf.intel.com>
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
minor naming correction to maintain consistency
ad58bd8 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@pre-commit-ci @adkakne
[pre-commit.ci] auto fixes from pre-commit.com hooks
c49ea84 
for more information, see https://pre-commit.ci

Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@ZePan110 @adkakne
Add hyperlinks and paths validation. (opea-project#132)
50d4167 
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
Merge branch 'main' into main
8b9e830
@adkakne
added support for older version of ragas
ecbe0d1 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
2db5af5 
for more information, see https://pre-commit.ci
@adkakne
Merge branch 'main' into main
7e35d0b
@adkakne
merge with OPEA main
6cb0d6a 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
testing automatic validation of ragas metrics
1047133 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
removing summarization_score metric
1a9d3e8 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
76790f3 
for more information, see https://pre-commit.ci
@adkakne
upgrading ragas from 0.1.16 to 0.1.19
8f0b59b 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
Merge remote-tracking branch 'origin/main'
82ef382
@lvliang-intel
Merge branch 'main' into main
a2bd899
@adkakne
Merge remote-tracking branch 'upstream/main'
d5c1da4
@adkakne
adding annotation free RAG assessment
1c0a0f4 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
improved README
0cfcebb 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
adding jsonlines to requirements
8c86ae9 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
lkk12014402
lkk12014402 reviewed Oct 11, 2024
View reviewed changes
tests/test_ragaaf.py Outdated Show resolved Hide resolved
adkakne and others added 5 commits October 10, 2024 23:53
@adkakne
fulfilled feature request - allow unit test case for RAGAAF
052431d 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
ccc864b 
for more information, see https://pre-commit.ci
@adkakne
removing extra inputs from evaluation
045eed6 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
Merge branch 'ragaaf' of https://github.com/adkakne/GenAIEval into ra…
348e947 
…gaaf
@adkakne
correcting class name for unit test
64493a4 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
Copy link
Contributor Author

adkakne commented Oct 11, 2024

Test is working with local endpoint for mistralai/Mixtral-8x7B-Instruct-v0.1 sharded on 8 HPUs.
image

@adkakne
test needs local ID
c58f377 
Signed-off-by: aasavari <aasavari.dhananjay.kakne@intel.com>
@adkakne
Copy link
Contributor Author

adkakne commented Oct 11, 2024

Hi @XinyuYe-Intel @lkk12014402 @minmin-intel,

After your review, I made updates to RAGAAF codebase, our final offering is -

  1. Data - can accept unit test cases, local datasets or benchmarking datasets
  2. Models - can accept all OpenAI models or a endpoint.
  3. Hardware - endpoint can run on HPU/GPU/CPU, local testing supports GPU/CPU, OpenAI testing uses their remote server.
  4. Metrics - we offer factualness, correctness, relevance and readability. User can add their own metrics easily.

Thank you for your final review and your insights for improvements in the code.

@lkk12014402
Copy link
Collaborator

LGTM

@XinyuYe-Intel XinyuYe-Intel merged commit 2413e70 into opea-project:main Oct 15, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XinyuYe-Intel XinyuYe-Intel XinyuYe-Intel approved these changes

@lkk12014402 lkk12014402 lkk12014402 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@adkakne @lkk12014402 @XinyuYe-Intel @daisy-ycguo @ZePan110 @lvliang-intel